Damage behaviors of plain concrete (PC), steel fiber reinforced concrete (SFRC), steel fiber reinforced and polymer modified concrete (SFRPMC) are studied in this paper by use of a Split Hopkinson Pressure Bar (SHPB). Three kinds of concrete materials appear obvious strain rate strengthening effects. SFRPMC appears a better resistance and energy absorption ability. A rate-dependent damage model is suggested to depict the impact damage evolution of three kinds of materials under different impact velocities. The simulation results showed the theoretical model could well describe the dynamic behaviors of the three kinds of materials, and steel fibers attribute more to resist crack develop in early stage, “bridge effect” of steel fibers slow up the damage evolution in SFRC, with the addition of polymer, the internal structures of SFRPMC were modified, SFRPMC gains better ductility, and appears a kind of “softening effect”, which makes the damage in SFRPMC develop more slowly than that in PC and SFRC under impact loading.