For the instability of ball-end milling cutter in high speed milling, instantaneous cutting process of high speed milling hardened steel was studied. The model of instantaneous cutting layer parameters of high speed ball-end cutter was established, and the influence of cutting speed and inclination angles on instantaneous cutting layer parameters were obtained. Using the model, instantaneous cutting force was studied, and high speed milling experiment was processed. Results show that the increase of cutting speed makes the change rate of cutting layer parameters increasing, leads to the energy concentration in cutting process, and increases the impact on milling cutter. The increase of inclination angle makes the instantaneous cutting layer parameters show a trend of decrease and the decrease of cutting thickness more rapidly, which caused instantaneous unit cutting force to increase and the instantaneous main cutting force appears increasing trend, and the cutting process become unstable.