The paper is devoted to vibration monitoring of rotating tools in modern milling machines. Dynamic analysis of slender ball-end milling process was performed and dynamics of the controlled structure was described. Instantaneous change in the spindle speed is applied in order to reduce vibration level. The method of vibration monitoring by means of spindle speed optimal-linear control was developed and implemented with success. Vibration monitoring during high speed milling was performed on the basis of results of computer simulation. These results were verified during experimental investigation on the Alcera Gambin 120CR milling machine.