A binary second order rational polynomial is adopted to simulate and extend the phase distribution on reference plane. In order to get the coefficients of the polynomial accurately, iterative least-square method based on the first order Taylor series expansion is used. The effect of the real reference plane profile error on measuring result is reduced by using extended unwrapped phase to substitute the original unwrapped phase. The effects of the random phase error and the system geometrical parameter error are decreased. The measuring accuracy of the system is improved. The principle of the 3D profile measuring system based on grating projection, the theoretic analysis for accurate estimate of phase distribution on reference plane, and the experimental results are presented.