Shot boundary detection (SBD) is the first step which segments video data into elementary shots for content-based video retrieval. In this paper, a shot boundary detection algorithm based on support vector machine (SVM) and particle swarm optimization (PSO) is proposed. First of all, the extracted features of pixel domain and compressed domain are combined to form a multi-dimension feature vector by using the scheme of sliding window. Next, particle swarm optimization with global search capacity is adopted to seek the approximately optimal parameters of radial basis function of SVM. Finally the model trained by the parameters obtained is applied to judge and categorize the frames into cut transitions, gradual transitions and non-transitions. The experimental results on the TREC video set 2001 demonstrate our algorithm is efficient and robust, and it solves the difficulty in parameter selection of SVM well.