In this work, natural convection heat transfer of nanofluids in a circular-wavy cavity is studied. The governing equations for the studied problem are presented based on the previous work and solved by finite element method (FEM). Because shape of the wavy wall has a significant effect on the heat transfer, it is important to find the optimal geometry for the wavy wall. In our study a novel response surface methodology (RSM) is applied for the optimization. This result indicates that in our case, amplitude has more significant effect on heat transfer coefficient than undulation number for a wavy wall. After finding the optimal geometry of wavy-wall for the cavity, effect of the constant coefficients (such as Ra, Le, Nr, etc.) on the Nusselt number, isotherm and streamlines and also nanoparticles volume fraction in the cavity are studied and discussed. The results show that the effect of Le number on the Nusselt number can be considered as significant only in the higher Nr numbers, while in lower Nr numbers, its effect is approximately negligible. The proposed method, FEM combined with RSM, is supposed to have widely application for the time-efficient optimization of heat transfer through irregular geometry. (C) 2016 Elsevier Ltd. All rights reserved.