This paper is concerned with the problem of automatic inspection of hot-rolled plate surface using machine vision. An automated visual inspection (AVI) system has been developed to take images of external hot-rolled plate surfaces and the detailed characteristics of the sensor system which include the illumination and digital camera are described. An intelligent surface defect detection paradigm based on morphology is proposed to detect structural defects on plate surfaces. The proposed method has been implemented and tested on a number of hot-rolled plate surfaces. The results suggest that the method can provide an accurate identification to the defects and can be developed into a commercial visual inspection system.