Based on Fick’s second law and the damage evolution due to sulfate attack, a new model is proposed to analyze the diffusion of sulfate ions in concrete. The relation between erosion damage and erosion time, as well as the concentration of sulfate ions is firstly investigated by virtue of the ultrasonic experimental results. Furthermore, the damage evolution is treated as the increase of porosity, and a new nonlinear differential equation on the diffusion of sulfate ions is established by substituting such an increasing porosity into Fick’s law. The nonlinear diffusion equation is then solved by numerical method. It is found that the erosion damage can significantly affect the diffusion of sulfate ions in concrete.