Ultra-precision raster milling is an emerging manufacturing technology for the fabrication of high precision and high quality components with a surface roughness of less than 10 nm and a form error of less than 0.2 μm without the need for any subsequent post polishing. Surface quality of a raster milled surface is affected by process factors and material factors, respectively. The process factors involve cutting conditions, cutting strategies, and relative vibration between the tool and the workpiece which are related to the cutting geometry and the dynamic characteristics of the cutting process. The material factors considered are material property and swelling of the work materials. Due to different cutting mechanics, the process factors affecting the surface quality are more complicated, as compared with ultra-precision diamond turning, such as swing distance and step distance. This paper presents an experimental investigation of the distinctive process factors affecting the surface roughness in ultra-precision multi-axis raster milling. Experimental results indicate that the influence due to the process factors can be minimized through a proper selection of operational settings and better control of dynamic characteristics of the machine.