Nowadays, face detection and recognition have gained importance in security and information access. In this paper, an efficient method of face detection based on skin color segmentation and Support Vector Machine(SVM) is proposed. Firstly, segmenting image using color model to filter candidate faces roughly; And then Eye-analogue segments at a given scale are discovered by finding regions which are darker than their neighborhoods to filter candidate faces farther; at the end, SVM classifier is used to detect face feature in the test image, SVM has great performance in classification task. Our tests in this paper are based on MIT face database. The experimental results demonstrate that the proposed method is encouraging with a successful detection rate.