Considering helical gear transmission's economic performance and drive reliability, construct multi-objective optimization model of the helical gear transmission with taking normal module, teeth number of small helical gear, helix angle and the gear width coefficient as design variables and taking the volume of small and large helical gear and opposite number of overlap ratio as objective functions. Propose multi-objective optimization design method based on coalition cooperative game theory where the two design goals are seen as two game players. By calculating the impact factor of design variables to objective functions and fuzzy clustering, the design variables are divided into strategy space of game players. Each game player takes its own revenue function as target and does single objective optimization in its own strategy space in order to get its own best strategy. The best strategies of all players form a combination of one round game and the optimal solution can be obtained through several game rounds. Example results show the effectiveness of game method.