In order to improve the retrieval speed and precision of images, the improved algorithm of extraction of image color features based on the both RGB and HSV color models was proposed in this paper. The algorithm can remove the repetitious vectors of compost in quantization process. While evenly quantizing model space, we can bring the compression of dimensions of image color features into full play and guarantee not to lose the main components of color features for color image. Then using RBF neural network and incorporating the values of color features, the image retrieval can be performed. The experimental results show that the data model of image color features in precise and high effective terms can be established and described by this algorithm, and the satisfactory results are obtained by applying the algorithm to the recognition of metallographic pictures for metallic materials. In addition, the algorithm can also be used in the evaluation of the discoloration of metallic casting alloys.