This study presents a smart remote controller (SRC) framework for the Android. The Android mobile device acts as the client side of the proposed SRC software. The software uses intuitive dynamic user operation modes to send remote control commands to the controlled side by leveraging the multi-touch events, gesture recognition and hand gesture features of the Android device. The remote controlled server side is based on a Java framework. This facilities portability to PCs or networked information appliances such as Internet TVs, thus, allowing users to establish connections and translate events to control corresponding programs or actions. In this design of the proposed SRC, advanced features are categorized into various modes that can be applied to the scenarios offices and digital homes.