The optimization of MgO-water nanofluids in order to reduce the cost and increase the heat transfer coefficient is investigated in this study. At first, the heat transfer coefficient is obtained at various values of solid volume fractions, diameters of nanoparticles, and Reynolds numbers based on empirical data. The cost amount is also determined in terms of solid volume fractions and diameters of nanoparticles. Then, the heat transfer coefficient function and the cost function are attained via RSM (Response Surface Method) and with a regression coefficient of over 0.997. The optimization is performed by the non dominated sorting genetic algorithm which has a significant capability of achieving optimal response. Finally, the Pareto front, the optimal heat transfer coefficient, and their corresponding minimum cost have been obtained. An appropriate correlation is also provided to achieve the optimal model of the minimum cost in terms of the maximum heat transfer coefficient. Optimization results have shown that, compared to the first optimization, the cost has decreased about 38% in the best case. (C) 2017 Elsevier Ltd. All rights reserved.