In this paper, the visual tracking algorithm for a moving target is proposed for the biped robot of which camera movement is irregular. Hexagonal Matching Algorithm is used to measure the changes of size, location, and rotation angle for a moving object from its image frame. For enhancing the efficiency of the tracking, we can adaptively adjust the starting point and the size of search area from the image information obtained. Finally, by using Affine Transform and Kalman Filter, the position estimation of the moving target is refined against the swing of the camera. Experiments with 20-DOF biped robot using mono vision sensor are implemented to prove the reliability of the proposed method.