With the rapid development of modern science and technology, the importance that control theory will be applied to mechanical engineering is increasingly evident. Matlab is a high-performance language for technical computing and Simulink provides a graphical user interface for building models as block diagrams. This article analyzes the application of the simulation software in mechanical engineering by taking position control servo system of CNC machine tool for example. Matlab also provides a function for frequency domain analysis, which has some functions of directly computing frequency response of systems, also drawing of some typical frequency response curves of the control system, and greatly simplifies the process of frequency-domain analysis. System stability can be judged from system frequency characteristic curve. According to judgment of stability of the system, characteristics of the system is easy to be observed by the simulation results, it can greatly improve quality and reliability of the programming with a friendly graphical user interface.