Digital sport video segmentation is an active area of research. And such an efficient sport video segmentation system is necessary. The system employs the feature-based motion estimator to estimate the video motion model parameters. Thus, we can use the result to compose the panorama. Project frames with motion parameters, the corresponding frames were connected and then stitched into a panoramic image according to infer the frames on a 2D manifold. For accurate alignment, we iterate between the motion estimator and topology determination to optimized parameters. Accuracy panorama composition is implemented to redress the error accumulation and achieve the pixel combination. After that, for the purpose of removing the foreground objects, we use the panorama estimation process. At last, we determine the sportsman segmentation masks that are the final output of the segmentation system. The test results showed that the proposed system is efficient.