In this paper, a novel calibration method is developed to improve the measurement accuracy of 3-DOF measurement system. The squareness error between three sensors and misalignment error with respect to reference coordinate of machine tool are calibration parameters. To estimate these parameters, reference ball is used and moved in the measuring ranges of 3-DOF measurement system. The relation between calibration parameters, position of reference ball, measured data of sensors are defined using geometric constraint and estimated using least square method. Finally, simulation is done to check the feasibility of developed calibration method. The result of simulation revealed the validation of developed method.