Nanocrystalline surface layer was fabricated on a quenched and tempered Cr-Si alloy steel by using Surface mechanical treatment. The microstructure features of various sections in the surface layer were characterized by using transmission electron microscopy (TEM). By analyzing the microstructural characteristic at different depths in the treated surface, the effect of the initial microstructures on grain refinement process of quenched and tempered steel was investigated. Experimental evidence showed the initial subgrains with small angle boundary and lower dislocation density were firstly developed into Lamellar-type dislocation cells (DCs) with dense dislocation walls (DDWs). Some initial subboundaries were moved to DDWs by dislocation activities. The width of lamellar-type DCs was 2-3 times of that of initial lathy subgrains. The size of the DCs and subgrains formed in the interim of refinement process was not uniformity. On the top surface the cementite granules were decomposed or fragmented to hyperfine particles, and the size of the grains tended to uniformity. Experimental analysis indicated the configuration of microstructure was affect by the initial microstructure in the initial stage and the interim of the grain refinement process. Surface nanocrystallization of Cr-Si steel can be attributed to dislocation activities.