The moving object detection of the video image is the basis of sequence image analysis, and it is the research hot issue of today’s foreign and domestic scholars. For detecting the moving object from the scene image in time, a detection algorithm of video moving object based on Gaussian mixture models is proposed in the paper. The pixel values are seen as the combination of the foreground Gaussian distribution and the background Gaussian distribution, and the background estimation and the adaptive background update will be put up. The statistical number of the foreground pixel of the current frame determines whether the light has a larger change, and it combines with the frame-difference method to detect moving object. The experimental results show that the algorithm can quickly and accurately establish the background model and accurately segment the foreground object.