Achieving the detection and tracking of moving targets has been widely applied in all fields of today's society. Because of the shortcomings of traditional video tracking system, this paper proposes a novel method for designing video processing system based on hardware design of FPGA and DSP, and moving target in video can be detected and tracked by this system. In this system, DSP as the core of the system, it mainly completes the processing algorithms of video and image data, FPGA as a coprocessor, responsible for the completion of the processing of external data and logic. The hardware structure, link configuration, program code and other aspects of system are optimized. Finally, through the experiment, the input frame rate of video is 40frames/s, and the image resolution is 512pixels × 512pixels, median 16bites quantitative image sequence, the system can complete the relevant real-time detection and tracking algorithm and extract targets position of image sequences correctly. The results show that the advantage is that this system has powerful operation speed, real time, high accuracy and stability.