In order to reduce the random noise influence on empirical mode decomposition (EMD), the original data is adaptively denoised by lifting wavelet transform to strain mode aliasing, avoid the pseudo mode functions and improve the quality in EMD. The method is employed to analyze the rotor oil whirl vibration signal. Obtaining intrinsic mode functions (IMFs), the instantaneous frequency and amplitude can be calculated by Hilbert transform. Hilbert marginal spectrum can exactly provide the energy distribution of the signal with the change of instantaneous frequency. Thus, the characteristics information of the rotor oil whirl vibration signal can be extracted effectively. Experimental result demonstrate the validity of the proposed method.