In this paper, local variance is used to describe the structural information of a color image in order to assess its quality. The representation method is different from conventional models in that some information that is sensitive to human eyes is enhanced by using local variance distribution. It encodes the local variance distribution of different channels of a color image into the three imaginary parts of a quaternion. The distance between the singular value feature vectors of the source image block and the distorted image block which are described by quaternion matrices is calculated. The experimental results show that the assessment results of the proposed assessment method are more consistent with the Human Visual System than those of the conventional assessment methods.