In order to minimize the fluctuation of gear transmission error (GTE) about the planetary gear transmission. A method was developed to deciding tooth profile modification curves of planetary transmission. According to the condition of the invariable design load, computing the dynamic characteristics of the planetary transmission system under modified and un-modified gear. At the same time, the compare is carried through of the dynamic characteristics for modified and un-modified gear. The results of the dynamic calculation indicate that the profile modification method can make the amplitudes of gear mesh stiffness change calmness and reduce the amplitudes of gear mesh stiffness by this method in paper. At last, the conclusion can be obtained that the tooth profile modification can reduce the vibration and noise of the planetary transmission system.