Rough end mill tools with serrated cutting edges profile are extensively used for removing a bulk of material and suppressing the chatter vibrations during milling operations. The serrated profile of cutting edge has phase shift from one flute to the next and interfere with the regeneration of waviness of the cut surface. In this research, serrated cutting edges are analytically defined and geometrically modeled as a NURBS curve. In addition; the chip load on the serrated cutting edges is calculated by a newly proposed algorithm. The validity of the algorithm is investigated by applying solid modeling techniques using ACIS 3D solid Modeler. In order to validate the results, several experiments were performed by utilizing serrated end mill. Verification of simulated and experimental results shows that the developed algorithm can be effectively implemented to simulate the cutting force for any geometry of milling tools as well as serrated end mills.