In this article, the nonlinear dynamic responses of drilling shaft system with hydrodynamic forces of cutting fluid were analyzed in deep slot hole drilling. A numerical method is presented to observe the states of the drilling shaft system. Using the proposed method, the periodic orbit of the drilling shaft motion and its period are calculated when the design parameters of drilling shaft system are subject to change, then the stability for dynamic responses of the drilling shaft system can be determined by the Floquet theory. According to the physical character of cutting fluid, the variational constraint approach is introduced to continuously revise the variational form of Reynolds equation at every step of iteration process. The nonlinear hydrodynamic forces of cutting fluid and their Jacobian are solved simultaneously without the increasing of computing efforts. The numerical examples show that the scheme of this study saves computing efforts but also is good precision, and can make a good reference for the dynamic design of drilling shaft system in deep slot hole drilling.