While rigid or pinned assumptions are adopted for steel frames in traditional modeling via finite element method, the actual behavior of the connections is usually neither. Semi-rigid joints enable connections to be modeled as partially restrained, which improves the quality of the model. To identify the connection stiffness and update the FE model, a newly-developed cross modal strain energy (CMSE) method is extended to incorporate the connection stiffness estimation. To illustrate the capability of the proposed parameter estimation and model updating algorithm, a four-story frame structure is demonstrated in the numerical studies. Several cases including Semi-rigid joint(s) on single connection and on multi-connections, are investigated. Numerical results indicate that an excellent updating is achievable and the connection stiffness can be estimated by CMSE method.