On the basis of the relation between force and deformation when the plastic deformation of soil mass is studied, nonlinear vibration roller model is built. Based on one type vibratory rollers select the system parameters and calculate the natural frequency. And according to the selected numerical value proceed the numerical simulation with different excitation force frequencies. Meanwhile, obtain and analyze the experimental data according to the vibratory roller experiment. Then the system response characteristic of nonlinear vibration roller is obtained, and the availability of system model is checked. All above provide the valuable theoretical basis for the research of vibrating compacting.