It has been necessary to adopting automatic detection of steel ball based on machine vision in industrial processing. The detecting instrument for surface quality of steel ball based on machine vision and embedded control technique and is applied to detecting external bug region of steel ball in bearing. Its image detection and control system require excellent real-time character and high control accuracy. This paper put forward a new design for image processing and control system of detecting instrument. Firstly, we adopted OTSU segmentation arithmetic based on image enhancement and median filtering. As a result, steel ball was detected by area character parameter. Secondly, we adopted TMS320LF2407A as main processor and integrated CPLD to develop an embedded controller. It presents a novel optimal design method for PID controller based on the ant colony optimization (ACO) algorithm to optimize traditional PID control algorithm. At last, a performance study of experimental system is conducted and it shows that the proposed method can be used in online steel ball detection applications.