Internal combustion Rankine cycle engine uses oxygen instead of air as oxidant during the combustion process in gasoline engine. Recycled fluid is employed to control the reaction rate and recycles the exhaust heat inside the cylinder as well. CO2 could be recaptured after separated from the exhaust gas (CO2 and water vapor) during condensation, and an ultra-low emission working cycle is achieved. Considering the side effects of water injection process, EGR is employed to control the combustion process and thermal efficiency of the oxy-fuel combustion cycle is calculated and optimized in this paper. Results show that the application of EGR could slow down the combustion process effectively, and appropriate EGR rate matched with ignition timing would control the reaction rate and cylinder pressure, therefore enhance the engine performance.