The uneven illumination distribution would affect the accuracy and stability of an optical multi-touch system. In this paper, different self-adaptive methods based on the type of interference source are integrated. When the ambient lighting changes, the luminance of the captured image will changes accordingly. This method tests the type of interference source by the speed and the region of ambient lighting change, and uses different function to update the background image rapidly. Experimental results show that with the help of the presented solutions, the ghost points are efficiently reduced and the system can adapt to the changes of ambient lighting in only 1 second.