In this work, the performance of an ejector refrigeration system using nano-refrigerants is investigated. A new hypothesis is proposed for flow boiling modeling, where nanoparticles are assumed to not migrate to the vapor phase as phase changes occur continuously; this causes a significant increase in nanoparticle mass fraction for high vapor quality values. This assumption shows a reasonable correlation with previously published data for R113/CuO mixtures, where an average deviation of 9.24% was obtained. A parametric analysis is performed to investigate the variation in heat transfer coefficient (HTC) with temperature, nanoparticle type, size, and mass fraction. Finally, the effect of nanoparticles on the coefficient of performance (COP) of the ejector refrigeration cycle as a response to the augmented flow boiling HTC is investigated by simulating a 5-kW cooling refrigeration cycle. Considering the advantage of using nano-refrigerants, a higher quality vapor was attained at the evaporator exit, resulting in an increase in the enthalpy difference in the evaporator in the ejector cooling cycle. As a result, a lower mass flux inside the evaporator is required to attain the same cooling capacity. Improved refrigerant-side HTCs improve the overall HTCs of the evaporator, allowing the evaporator to operate at lower temperature differences and higher pressures, which consequently increases the cycle COPs. Furthermore, the refrigerant vapor quality increases at the evaporator exit, leading to an enhanced COP of the cycle. The augmentation in COP reached 24.7% and 12.61% for R134a with 2 wt.% CuO and Al2O3, respectively, whereas the vapor quality for the refrigerant leaving the evaporator increased from 0.7616 for the case of the pure refrigerant to 0.8212 for R134a/CuO 2 wt.% nano-refrigerant.