This paper presents a template-based vision system to detect and classify the nonuniformaties that appear on the semiconductor wafer surfaces. Design goals include detection of flaws and correlation of defect features based on semiconductor industry expert’s knowledge. The die pattern is generated and kept as the reference beforehand from the experts in the semiconductor industry. The system is capable of identifying the defects on the wafers after die sawing. Each unique defect structure is defined as an object. Objects are grouped into user-defined categories such as chipping, metallization peel off, silicon dust contamination, etc., after die sawing and micro-crack, scratch, ink dot being washed off, bridging, etc., from the wafer. This paper also describes the vision system in terms of its hardware modules, as well as the image processing algorithms utilized to perform the functions.