Three dimensional molecular dynamics simulation on the nanocutting of monocrystalline silicon is carried out to investigate the material deformation behaviors and atomic motion characteristics of the machined workpiece. A deformation criterion is developed to determine the material deformation and phase transformation behavior in the subsurface layer based on the single-atom potential energy variations. The results show that the machined chips suffer a complex phase transformation and eventually present an amorphous structure caused by the plastic deformation behavior. A polycrystalline structure is obtained on the machined surface. Both plastic and elastic deformation simultaneously takes place on the machined surface, and elastic deformation takes place under the machined surface. In order to further unveil the mechanism of nanocutting process, the displacements of all atoms are also simulated. The simulation results shows that different atomic motions occur in different regions in the workpiece, and the chips formations occur via extrusion.