In this paper, the analytical formulations among the damping factor and the other parameters are studied by Melnikov method for the classic second-order model of power system. The proximity of Melnikov method is analyzed by comparison between the obtained critical damping factor with the result by time-domain simulation method. It points out that this method exist great error in the range of engineering application. To solve this problem, an improved method is proposed to calculate the critical damping factor based on RBFNN. Finally, simulations show that it can effectively improve the calculation accuracy.