Different degrees of freedom exist in amorphous matter, and are at the origin of relaxation processes (e.g. structural, mechanical). The effect of physical aging of the glassy network on secondary and primary (and cooperative) relaxation is studied. It is shown that if the temperature of aging is below the liquid glass transition temperature and above that of the secondary relaxation, the strength of this last relaxation changes in a non monotonous complex way while the characteristic time of the primary relaxation increases. This feature is explained taking into account the heterogeneous nature of the glassy network. This concept is in agreement with recent inelastic light scattering observations showing that fluctuations of the elastic constants at the nanometric scale are characteristic of the glassy state.