Cold rotary forging is an advanced but very complex incremental metal forming technology with multi-factors coupling interactive effects. The position between the upper die and the workpiece has a significant effect on the cold rotary forging process. In the current work, a 3D elastic-plastic dynamic explicit FE model of cold rotary forging of a cylindrical workpiece is developed under the ABAQUS software environment and its validity has been verified experimentally. On the basis of this reliable 3D FE model, the effects of the position between the upper die and the workpiece on the cold rotary forging process have been thoroughly revealed. The results show that with increasing the distance between the pivot point of the upper die and the centre of the workpiece, the deformation of the workpiece becomes more inhomogeneous and the maximum axial forging force and forging moment gradually increase. The results of this research not only provide valuable guidelines for the installation and adjustment of dies in the cold rotary forging process, but also help to better understand the deformation mechanisms of cold rotary forging.