In this paper, we propose an interactive remote control system using image processing technique for three dimensional displays. In order to control three dimensional displays, we present an algorithm based on gestures recognition. Then we adopt cell-based design flow and 0.18μm TSMC single poly and six metals processing to perform the system chip. Besides, the SMIMS FPGA development board is used to achieve the system. The results prove the performance reaching the well quality of interactive remote control system for three dimensional displays.