For a long time, the mineral additive was regarded as inert filler which had no reaction with the asphalt, so does the concrete mix. But some research conducted in the recent year proved that was not the case. In this study, four types of mineral materials (diatomite, kaolin (China Clay), Na-bentonite and Ca-bentonite) were selected out to investigate their effect on the asphalt mastics. The high temperature performance, low temperature performance and the temperature susceptibility of the asphalt mastics were evaluated. The result showed the diatomite rank first of the four minerals. The porous structure, huge specific area, mineralogical surface characteristic and the irregularity might be the reason for its positive effect. The bentonite, especially Na-bentonite is unfavorable to the asphalt modification.