In NC machining, the precision of the final parts is affected by many factors, including geometric deviation of machine tool components and structures, deformation of process system caused by cutting force and cutting heat, servo delay, tool wear and so on. Among which geometric error of machine tool is one of the most important factors. This study focused on geometric error identification and compensation of rotation axes of five-axis machine tools. A new method was proposed to identify the 6 geometric error parameters of each rotation axes of five-axis machine tools based on a ball-bar system. Regarding the machine tool as a rigid multi-body system (MBS), a geometric error model was established based on homogeneous transfer matrix (HTM). Finally, the geometric errors were compensated by correcting NC codes by the prototype software system developed in this study. An experiment and an application were conducted and the results show that the proposed method is effective to improve the machining accuracy.