For the problem that manual adjustment of the parameters of controller in sensorless control system costs too much time, manpower and always can not get a good result, a new method based on improved particle swarm optimization algorithm is proposed to optimize the parameters. The improved algorithm is based on the standard particle swarm optimization with the simulated annealing algorithm and chaotic search brought in. The speed of motor is estimated by the extend Kalman filter. The error between measured speed and estimated speed of the permanent magnet synchronous motor rotor is used as the fitness function in order that the parameters in the covariance matrix is adjusted.The result of simulation indicates that high estimation precision can be got and the motor represents steadily with few of ripple of the actual speed.With this method, the time of adjustment is reduced and manpower is saved. In addition, the validity of the method is proved in experiment with dSPACE.