The nonlinear chatter in grinding machine system is discussed analytically in the paper. In higher speed grinding process, the self-excited chatter vibration is mostly induced by the change of grinding speed and grinding wheel shape. Here the grinding machine tool is viewed as a nonlinear multi-D.O.F. autonomous system, in which hysteretic factors of contact surfaces are also introduced. Firstly, the DOFs of the above system are reduced efficiently without changing its dynamic properties by utilizing the center manifold theorem and averaging method. Then, a low dimensional system and corresponding averaging equations are obtained. The stability and bifurcation of chatter system are discussed on the base of deduced averaging equations. It is proved that chatter occurs as a Hopf bifurcation emerging from the steady state at the origin of system. The theoretical analyses on the multi-DOF chattering system will lead to further understanding of the nonlinear mechanisms of higher speed grinding processes.