Developments of tetrahedral amorphous carbon (ta-C) films having low residual compressive stress are essential to extend the applicability of the films. The annealing of the ta-C films was known to be an effective way for the reduction the stress of the films. However, the effects of annealing on the atomic structure of ta-C films have not been fully understood. The atomic structure changes by the annealing were studied using molecular dynamics simulation. The simulation showed that the annealing caused an increase of the atomic volume of ta-C film, which explained the stress reduction partially. However, the tendency of the stress reduction was different to high and low stress films. The annealing substantially reduced the stresses of high stress films compared to those of low stress films. Atomic structure analysis showed that the reason for the asymmetric stress reduction resulted from the relaxation of highly distorted bonds that existed in as-deposited films.