Rotor steel has been widely used in marine engineering and power industry because of its high yield stress and anti-fatigue performance at high temperature. The surface integrity must be guaranteed due to severe working environment. Hence it is worth studying residual stress (RS) for its significant influence on the quality of machined part. In this paper, an effective model is built with the help of finite element method (FEM) to evaluate RS induced by machining of rotor steel 26NiCrMoV145. An adaptive mesh control is introduced in this work to avoid excessive element distortion. Johnson-Cook flow stress model is proposed to model workpiece material. Rolling process is necessary to obtain desired RS according to the performance of form-milling process. In order to validate FEM predictions, both milling and rolling experiments have been carried out. The predicted RS profiles are in reasonable agreement with the experimental ones.