Local image representation based natural image classification is an important task. SIFT descriptors and bag-of-visterm (BOV)method have achieved very good results. Many studies focused on improving the representation of the image, and then use the support vector machine to classify and identify the image category. However, due to support vector machine its own characteristics, it shows inflexible and slower convergence rate for large samples，with the selection of parameters influencing the results for the algorithm very much. Therefore, this paper will use the improved support vector machine algorithm be based on ant colony algorithm in classification step. The method adopt dense SIFT descriptors to describe image features and then use two levels BOV method to obtain the image representation. In recognition step, we use the support vector machine as a classifier but ant colony optimization method is used to selects kernel function parameter and soft margin constant C penalty parameter. Experiment results show that this solution determined the parameter automatically without trial and error and improved performance on natural image classification tasks.