A new matrix on the covariance of covariance is formed from the auto/cross-correlation function of acceleration responses of a structure under white noise ambient excitation. The components of the covariance matrix are proved to be function of the modal parameters (modal frequency, mode shape and modal damping) of the structure. The number of vibration modes associated with components of the matrix is only limited by the sampling frequency. Compared to the general methods for extracting modal parameters, the formulated covariance matrix contains more information on the vibration modes of the structure. An in-service suspension bridge is analyzed. Only the output acceleration responses are used to compute the covariance of covariance (CoC) matrix and the Pattern Assurance Criterion (PAC) values. From the CoCs variation and the changing trend of the PAC curve, the healthy condition of the bridge is assessed. The method is independent of analytical model and huge amount of data can be analyzed. It is especially suitable for long-term on-line health monitoring of a structure.