A new approach is proposed to detect and track the moving object. The affine motion model and the non-parameter distribution model are utilized to represent the object firstly. Then the motion region of the object is detected by background difference while Kalman filter estimating its affine motion in next frame. Center association and mean shift are adopted to obtain the observation values. Finally, the distance variance and scale variance between the estimated and detected regions are used to fuse the observation values to acquire the measurement value. To correct fusion errors, the observable edges are employed. Experimental results show that the new method can successfully track the object under such case as merging, splitting, scale variation and scene noise.