By using the intrinsic character of the frame structures, a new direct damage identification method of the frame structures based on residual force vector is presented. The stiffness connectivity matrix is obtained by sensitivity analysis. The stiffness matrix perturbation of the damaged structure can be expanded by the stiffness connectivity matrix, then substituting it into residual force vector equation, the new residual force vector equation expressed by the stiffness connectivity matrix is obtained. This equation can be solved directly, so the stiffness perturbation of layers is got, by it, the damaged locations and the extents of the frame structures are identified. At last, a numerical example of a seven-layer frame structure is given, the identification result shows that, only with the first-order mode of the damaged structure, the damage can be identified accurately. Even in considering of some degreed noise of the mode parameters, the structure damage can be identified well, demonstrating the effectiveness of presented method.