The interactive approach to virtual system is a hot research topic. This paper presents a novel interactive approach based on hand gesture recognition. The hand gesture segmentation method is based on the adaptive background model. The method improves the ability of adaptation environment. And an improved algorithm of hand recognition is presented by using Fourier Descriptor and BP neural network. Experiment results indicate that the approach is flexible, realistic and exact.