We present a servo control model in a particle filter to realize robust visual object tracking. Particle filter has attracted much attention due to its robust tracking performance in cluttered environments. However, most methods are in the mode of moving object and stationary camera, as a result, the tracking will become failure if the object goes out of the field of view of the camera. In this paper, a closed loop control model based on speed regulation is proposed to drive the pan/tilt/zoom(PTZ) camera to keep the target in the center of the camera angle. The experiment result shows that our system can track the moving object well, and can always keep the object in the middle of the field of view. The system is computationally efficient and can run in real-time completely.