The roller gear cam surface is often machined by the unequal diameter manufacture method, which means the tool diameter is smaller than that of the roller and the tool position compensation method is most used. For tool position of roller gear cam, it is important to confirm the compensation vector for the tool position compensation method, including compensation value and direction. In the paper, a new self-adaptive tool position optimization method is proposed, which make minimizing the normal machining error as the object function and make two compensation factors as the optimization variables. This algorithm can find out the best compensation direction and value by the tool position optimization for any cam rotation angle and make the tool position self-adaptive and flexible compensation according to the machining error. A numerical calculation example shows that the optimization algorithm can feasibly reduce the machining error. At last a conclusion has been drawn that the radius difference between the tool and the roller is the best compensation value and the best compensation direction is not fixed.