In view of the inherent limitations of classical Hough transform in the detection of line, an improved algorithm of randomized Hough transform is offered in this paper. The feature point sets are segmented based on the connected component labels in this algorithm. The elements of various point sets are stored in sequence so as to reducing the invalid samples. By using a valid line detection area and a dynamic storage management, the temporary storage units are gradually reducing with the iterative process. The speed of the line detection and the accuracy of identification can be improved, the storage space can be reduced, the computational complexity can be cut down and some unnecessary calculations can be avoided by using this algorithm. However, it still has the characteristics of the classical Hough transform that it is not sensitive to noise. The line edge where the cutting tool is on can be pinpointed. To achieve the vision-based mode of tool wear state, it can accurately detect rake and flank face of the cutting tool, and can effectively remove the influence to the tool wear detection from the BUILT-UP EDGE in cutting process. By the validation of testing examples, the algorithm has high speed, low memory occupation and high accuracy.