Microstructural evolution and mechanical properties of the AZ31 magnesium alloy during rolling and annealing process were investigated. The sheet samples were prepared after different stages of the hot rolling, cold rolling and annealing processes. The hot rolling temperature was between 300-450C and the final thickness of the cold rolled sheets was 1.5mm. The cold rolled sheets were annealed at different annealing temperature (260-350C) for different time (10~120min). Tensile test was performed to investigate the mechanical properties of the samples obtained from different stages. With aid of the optical microscopy, scanning electron microscopy (SEM) techniques, the microstructure of the samples were characterized and the results were related to the mechanical properties. It was found the hot-rolled sheets exhibit higher ductility comparing with the cold-rolled sheets. The microstructural investigation showed that the microstructure of the hot-rolled samples was dominated by recrystallized equiaxed grains while the microstructure of the cold-rolled samples dominated by deformation twining. By applying annealing on the cold-rolled sheets, fine recrystallization grains were obtained and ductility of the samples was improved. The effects of the grain size and twining on mechanical properties of the AZ31 sheet were further discussed based on the experimental results.