Carburizing and quenching process conditions must be optimized to minimize quenching distortion and dispersion; this includes the parts racking design used for quenching. The racking positions of the gear determine the distortion of the tooth through the effect of the cooling condition on the surface. We analyzed gear tooth distortion on the carburizing-oil quenching process. The cooling dispersion at the gear surface was measured. The heat transfer coefficient was calculated by a design of experiment study, adaptive simulated annealing, and non-linear sequential programming on quadratic language using a cooling simulation model of the gear tooth under four quenching conditions. The coefficients were attached to the boundary condition of the gear model surface, and the gear distortion was investigated using a five-model simulation that used both a homogeneous model and the four heterogeneous surface cooling models. Evaluation of the differences in accuracy between these simulations showed that the shape accuracy in the heterogeneous simulations was better than in the homogeneous one. To calculate the distortion tendency accurately, we need to assign a detail cooling condition to the gear surface.