Manufactured miniature parts conventional machine tools are substituted by mini-scale manufacturing system called miniaturized machine tool (mMT). However, calibration of mMT is extremely difficult due to its small dimensions. This paper presents a novel optical measuring system to measure integrated geometric errors namely a) three translation errors and b) three rotational errors. The squareness error between two moving axes was calculated by least-square method. These 7-DOF geometric errors were acquired simultaneously in one setup for each motion direction. The proposed measuring system consisted of two laser diodes, two beam splitters and three 2D position sensing detectors (PSDs). Configuration of this measuring system was proved by homogenous transformation matrix (HTM), and the relationship between PSD readings and geometric errors calculated by the algorithm were derived according to the configuration. Simulation were carried out to prove the validity of this algorithm. Sensitivity analysis based on mounting errors of system components is also presented in this paper. It is useful for further experiments in alignment of system components.