This paper addresses digit-writing hand gesture recognition using ego-motion detection algorithm. In training stage, according to the manual labeled feature points, the ego-motion detection algorithm is constructed and the corresponding average feature is obtained. In recognition stage, the interesting digit-writing hand gesture region is firstly segmented by skin and movement cues. Secondly, the models are fitted to the image that includes the hand gesture, and the relative features are extracted. Thirdly, the classification is done by comparing the extracted features and average features.