An object tracking framework based on adaboost and Mean-Shift for image sequence was proposed in the manuscript. The object rectangle and scene rectangle in the initial image of the sequence were drawn and then, labeled the pixel data in the two rectangles with 1 and 0. Trained the adaboost classifier by the pixel data and the corresponding labels. The obtained classifier was improved to be a 5 class classifier and employed to classify the data in the same scene region of next image. The confidence map including 5 values was got. The Mean-Shift algorithm is performed in the confidence map area to get the final object position. The rectangles of object and background were moved to the new position. The object rectangle was zoomed by 5 percent to adapt the object scale changing. The process including drawing rectangle, training, classification, orientation and zooming would be repeated until the end of the image sequence. The experiments result showed that the proposed algorithm is efficient for nonrigid object orientation in the dynamic scene.