A dynamic model of the spindle-cutter coupling system is established including the gyroscopic effects and shear deformation. Dynamic characteristic of the system is analyzed by utilizing the whole transfer matrix method (WTMM) based on the Riccati transformation, and strain energy distribution of the system is estimated. Dynamic characteristic of the system is optimized by univariate search technique. And optimal path in the system optimization is provided. According to the optimal path, a method of local mode elimination by decreasing stickout length of the milling cutter is proposed. The calculation results indicate that preload level of front bearing and rear bearing is the key factor of dynamic optimization after local mode elimination.