A method of straightness error compensation is presented, which is used in ultra-precision machining with nano-scale accuracy for a large mandrel manufacture. A set of measurement system in situ is developed, in which an ultra-smooth glass-ceramic flatness gauge and a non-contact micro displacement sensor with nano-scale resolution were used as a reference and sensor to get the straightness error of machine tool movement. The real straightness error can be obtained after subtracting the surface profile of the gauge from the original straightness error curve. Based on the real straightness error data, a new NC program was made for compensating the error from the axis movement of machine tool. As a result, after straightness error compensation, the straightness errors of two axes of ultra-precision machine tool are 68nm/400mm and 54nm/300mm respectively.