Metals deformed to high and ultrahigh strains are characterized by a nanoscale microstructure, a large fraction of high angle boundaries and a high dislocation density. Another characteristic of such a microstructure is a large stored energy that combines elastic energy due to dislocations and boundary energy. Parameters of the deformed microstructure significantly affect annealing processes such as recovery and recrystallization. For example, the recovery rate can be significantly increased after high strain deformation and restoration may occur as either discontinuous recrystallization or structural coarsening. A characterization and analysis of deformed and annealed microstructures presented in this work covers Al, Ni, Cu and Fe heavily deformed by rolling, accumulative roll bonding (ARB), equal channel angular extrusion (ECAE) and high pressure torsion (HPT). The important effect of recovery on subsequent restoration processes is discussed along with the effect of heterogeneities both on the local scale and on the sample scale.