As a new type of high-intensity combustion technology, oxy-fuel combustion technology can effectively reduce the generation and emission of pollutants, particularly near zero emission of carbon dioxide when applied to the waste incineration process. Compared to the conventional air waste-to-energy incineration power generation, the municipal solid waste oxy-fuel combustion power generation system is more complex, resulting in a relatively large space for optimization. In this work, a waste-to-energy incineration power plant in Shenzhen, China, is taken as the original object, and used to establish the process simulation of the conventional plant using Aspen plus. The results are compared and verified with the operation data. Based on the results, models or subsystems are set up for the air separation unit, the municipal solid waste oxy-fuel combustion power system and the flue gas processing compression system, respectively. Then, the subsystems are coupled and connected to establish the whole process simulation of the waste oxy-fuel combustion power plant, and the optimization analyses of the overall plant operating parameters are presented. The results show that the best supplying oxygen concentration is 96%, the carbon dioxide recovery rate of the entire system is 96.24%, and near-zero carbon dioxide emission is basically achieved. The energy consumptions sharing by the flue gas processing compression subsystem, the air separation subsystem, and the others account for 19.82%, 54.29%, and 25.89% of the total energy consumption, respectively. After coupling optimization analysis, the net power generation efficiency of the municipal solid waste oxy-fuel combustion plant increases 2.69%, from 6.88% to 9.57%.