The nonlinear dynamic model of a gear-shaft-housing system is established by using finite element method (FEM). The nonlinear dynamic response of the coupling system under gear tooth time-varying stiffness, manufacturing error and other internal excitations is analyzed. Displacement, velocity and acceleration response of the coupling system and meshing stiffness and bearing support stiffness influences on vibration response of the coupling system are studied. The results show that the vibration amplitude near housing bearing seat is largest, secondly is the top of the housing and the vibration response at the sides and bottom of the housing are the smallest. As meshing stiffness and bearing support stiffness increase, vibration velocity and acceleration of the housing increase gradually. But bearing stiffness influence on the vibration response of the coupling system is significantly larger than that of meshing stiffness.