The screw motor is one kind of volume type power generator, the stator and the rotor is screw motor's core part. In this paper, cutting tool's internal stress and the distortion under the condition of milling inner helical curved surface was dissected with finite element method. Variable rule of feed engagement with changing of the processing parameter was discussed. The modal analysis of the special-purpose cutting tool was carried. To reduce the abnormal wear of special-purpose cutting tool in processing process and increase the special-purpose cutting life, the optimizing scheme was given. This research provides the theory reference for the optimized processing technological parameter.