The numerical approximation is proposed to explore the effective improvement of precision of position and velocity requirement in the mechanical transmission which hold the nonlinear gear system with clearance unit. The transfer function is adopted to compare the difference between the ideal model and model with the nonlinear gear system. The characteristics of the model with nonlinear gear system are detected by calculating the various parametric coefficients in the model. And then the numerical database is constructed to conveniently display the system performances, which also offers the plausible method to obtain optimum parametric coefficients using the numerical approximation. The tolerance equation between the approximate function and ideal function is set an objective function. The most optimum parameter can be obtained from the objective function by the optimization theory. The results show that these calculated optimum parametric coefficients can achieve the system stability, and improve the synthesis performance of control system. The proposed method in this study avoids to seeking for the approach to analyze the complex nonlinear system or build new control scheme, which is liable to use in the practical application. The simulative results demonstrate the effectiveness of the proposed method.