To handle occlusion and accumulation error in tracking procedure, a novel method is proposed. Firstly, the object feature is modeled with kernel based color histogram. Then, mean shift is used to localizing the object with Kalman filter providing initial iteration location and scale. Object observation value is acquired by weighting the similarities of hue and saturation in x and y-directions. Finally, occlusion and scene disturbance are judged by maximal similarity and the matching deviation, so as to selectively update the object model. To suppress the accumulation error, the noise covariance is updated according to the iteration error in the latest N frames. Experimental results show that the proposed method is robust in tracking the occluded objects under complex scene.