The microstructural evolution, the changes in microhardness and the recrystallization behavior of a modified 316L stainless steel were investigated during high pressure torsion (HPT) and subsequent annealing. To study the impact of the governing process parameters on the evolving microstructures, the applied strain, the strain path and the annealing temperatures were varied. In contrast to ordinary single phase steels, which showed a decrease in the structural size ending in a saturation of the microstructural refinement between an equivalent strain eq of 10 and 15, HPT of the modified 316L results in a steep increase in shear stress at very small strains and the saturation region is reached far before eq = 10. Studies using the transmission electron microscope (TEM) revealed that at large strains the original coarse grains are converted by the massive intersection and fragmentation of twins into a nanometer-scaled microstructure. In the case of monotonic HPT, shock annealing of the deformed discs results in rows of fine and coarse grains. In the cyclic deformed discs a homogenous, fine-grained and almost fully recrystallized microstructure was observed. The results clearly show that both the strength and ductility of the material can be significantly influenced by SPD and subsequent annealing. Possible reasons for the observed differences in the deformation and annealing behavior are discussed.