Vehicle classification and tracking is considered as one of the most challenging problems in the field of pattern recognition. In this paper, Particle Swarm Optimization (PSO) based method is exploited to recognize vehicle classes. Vehicle features, such as vehicle size, shape information, contour information are extracted. Each vehicle class is encoded as a centroid with multidimensional feature and PSO is employed to search the optimal position for each class centroid based on fitness function. After vehicle classification, an improved meanshift algorithm is presented for vehicle tracking. The algorithm’s evaluations on video image series, moving vehicle detection, vehicle classification and tracking are respectively conducted. The results show that PSO ensures a promising and stable performances in recognizing these vehicle classes, and the improved meanshift algorithm can achieve accuracy and real-time for tracking moving vehicles.