In order to improve the calculation method of the vacuum system design, the research focus on the way applying the Fluent software to calculation of gas flow in vacuum system in this paper. It is proved that the permission pressure lower limit is 1Pa for meeting the continuity hypothesis. The applicable flow patterns include turbulent flow, laminar flow and transitional flow between turbulent and laminar flow, which are totally defined as the viscous flow in the traditional vacuum field. By means of user define function (UDF), a layer mesh cell of negative quality source is defined to simulate the constant volume flow rate at the inlet of positive displacement vacuum pump, which makes up for the lack of compressible fluid velocity outlet boundary conditions in Fluent. The usability of Fluent in vacuum system is confirmed by a successful calculating example of gas flow in an ice condenser of vacuum freeze dryer.