The dynamic behavior of a ball-beam system is highly nonlinear and its characteristic is difficult to define. In this paper we present a new ball-beam balancing control system using machine vision to feedback the beam angle and ball position on the beam. Adaptive threshold based continuously mean shift vision tracking algorithm is applied to record the ball position and the beam angle with highly captured frame-rate. The proposed vision tracking algorithm is tolerant to lighting influence, highly computing efficiency and more robust than traditional template pattern matching or edge detection algorithm under non-ideal environment. The vision tracking performance is experimentally tested on a ball-beam benchmark system, where a PD controller is applied to control the motion of the ball to maintain balance. Experimental result shows that the beam angle measurement, ball tracking and balancing control of the vision feedback system are robust, accurate and highly efficient.