In this paper, the novel measurement system for estimating the six geometric error of rotary table has been proposed. This system configuration comprising four components namely laser diode, beam splitter, position sensitive detector (PSD) and deflecting mirror. The advantage of this approach is that all six geometric error of rotary table can be evaluated by just measuring four set of points from this system configuration. To estimate the geometric errors, a linear relationship is established mathematically between measurement points and geometric errors. The line equation and plane equation along with homogeneous transformation matrix are used to define the linear relationship. The mathematical model is simulated to verify the validity of the system; the result shows the proposed system is suitable to measure the geometric errors of rotary table.