In ultra high precision roundness measurement, a high accuracy and high integrated measuring circuit system equipped with inductive sensor are a developing trend. In order to meet the demand, a novel method to improve the accuracy of measuring circuit of amplitude modulation sensor using digital synthesizing, digital phase sensitivity detection (PSD), and digital set average techniques, is presented. The measuring circuit system based on the method is easy to realize and to operate. It can be plugged directly into an ISA Bus slot for use. Experimental results show that the stability of the sensor measuring system that consists of the measuring circuit system and a Taylor inductive sensor reaches nanometer accuracy. The digital measuring circuit of amplitude modulation can not only be applied widely in amplitude modulation measuring circuit for use of inductive sensors to improve the accuracy of the sensor measuring circuit system but also used in the one for use of capacitance sensors and magnetic grating sensors.