3D scanning probe is a kind of measuring instrument that is an indispensable part of Gear Measuring Center. It’s used for obtaining micro-displacement in three directions through the deformation of the three mutually vertical parallel springs. The accuracy of the probe system has critical influence on Gear Measuring Center. Because of the limitation of the structure of parallel spring, the installation errors of the probe and the influence of environmental factors, the parallel springs in three directions are not always strictly vertical, the interference and coupling among X,Y,Z directions exist. This problem is very difficult to be solved by adjusting the mechanical structure of 3D scanning probe system. However, the accuracy of probe system can be improved by adopting software-compensation through calibration. In the paper, the calibration matrix of the probe system is generated by fitting the points sampled on the standard ball with the method of the least square. The points are sampled through controlling the probe to touch standard ball in several given directions. The experimental results show that the measurement errors caused by the coupling among the three perpendicular directions can be effectively decreased by the coefficient matrix mentioned above. The method of calibration is practical and effective for improving the measurement accuracy of 3D scanning probe.