In this paper, we proposed a method to attain a high sensitivity of weak optical signal detection by using the multi-wavelength modulation principle. In this method, the optical phase carrying the measurand is obtained by averaging the phases obtained in various orders of heterodyne harmonics. The employment of higher heterodyne harmonics reduces the drift effect in optical detection. The proposed method is demonstrated on the nano-scale displacement measurement by using a single channel fiber interferometer. In the experiment, the displacement is simulated by moving a mirror with a piezoelectric tube. The results show that more repeatable measurements with a resolution of 35nm can be obtained by using the proposed method.