Gear profile modification is an important method for reducing system vibration and noise of a gear transmission. In this paper, three kinds of gear profile modification curves are studied, and relevant experiments are conducted. Some useful conclusions based on the theoretical analysis and experimental results have been obtained. Line, parabola and sinusoid modification curves are very useful in improving the dynamic performance of a gear transmission system by reducing system vibration and noise. In order to design the optimal tooth profile modification curve, the curve type must be decided firstly according to the specific working condition, and the curve parameters can be determined by means of optimum programming.