The interference-fit riveting technique is widely used in aircraft assembly, which can generate residual stress around the wall of hole and significantly improve the fatigue life of the structure. The residual stress is usually determined by experiment or numerical simulation, locking of theoretical support. So this paper researches the residual stress modeling of the panel in a riveting process. Firstly, based on the axisymmetric upsetting theory, dividing the riveting process into two stages, the formulas for calculating the contract pressure caused by the rivet around the hole of panel are built. Secondly, under the action of contact pressure, the residual stress distribution expressions are developed by elastic-plastic mechanics. Finally, in ABAQUS system, a three-dimensional instance made up of one flush rivet and two panels of aluminum alloy is used to verify the theoretical formulas. The feasibility of the theoretical formulas about the residual stress of the panel is validated by comparing the theoretical calculation and finite element simulation results.