A previously developed equation-based flowsheet optimization framework is extended and applied to design sub-ambient separation systems for oxy-fired coal power systems with carbon capture. Unlike most commercial flowsheet design and optimization tools, the proposed methods use exact derivatives and large-scale nonlinear programming algorithms to solve large flowsheet design problems with many degrees of freedom, including the simultaneous design of air separation units (ASUs) and their accompanying multistream heat exchangers. Emphasis is placed on additional model improvements regarding thermodynamic calculations. In order to maintain differentiability, complementarity constraints are used to model switches, including vanishing and reappearing phases. Nevertheless, these complementarity constraints may construct trivial phase equilibrium solutions, and a procedure based on embedded bubble and dew points calculations is proposed to avoid them. Furthermore, additional complementarity constraints for the cubic equation of state model are proposed to ensure correct phase identification in the supercritical region. Finally, the efficacy of these new models are demonstrated by optimization of the CO2 processing unit and compression train for an oxy-fired power plant. (C) 2015 Elsevier Ltd. All rights reserved.