The design of a kind of video processor with the function of auto-focusing is introduced in this paper. The video processor receives digital 8-bit YCbCr video signal. With a piece of DDR2 SDRAM as frame buffer, the video processor realizes the function of format conversion, frame up-scaling and color space conversion. The paper proposes an improved gray scale differential auto-focus algorithm and calculates the focus value of the current image real-timely. Applying the mountain climbing searching strategy to control the moving of the focus motor, and realize the function of auto-focusing. The whole design is realized in VHDL and verified with Xilinx XUPV5-LX110T FPGA developing board.