The real-time vehicle classification plays an important role in Intelligent Transportation System (ITS). How to effectively improve the accuracy rate and the speed of the vehicle classification is still a hot research issue, the classification algorithm has to be effective but simple. In this paper, a vehicle detection algorithm based on edge-based background difference and region-based background difference is proposed. This algorithm can extract the moving vehicle completely, eliminate vehicle shadow effectively, and it is still significant despite the variations of illumination and weather conditions. The algorithm is simple with low computation quantity and suitable for real-time system. In the feature extraction process, the feature vector can be obtained in short time. Support vector machine (SVM) is also discussed in the classification process. The experimental result shows that the system can accurately recognize the vehicles.