This paper aims to study the strain softening behavior of soft rock. A modified equation of unified strength theory is proposed that is convenient to be applied in geotechnical engineering where compression is customarily taken as positive. And also the limit line on deviatoric plane of this modified equation is derived and introduced into the three dimensional (3D) elastic viscoplastic constitutive model of Yin and Graham. Parameters of the model are determined from experiments of the diatom soft rock specimens. Numerical simulations are performed to compare the strain softening behavior predicted in this paper and triaxial experimental results. Simulation results show that the proposed model can accurately describe the strain softening of soft rock.