When heat transfer enhancing technology is applied to design heat transfer unit, the improvement of heat transfer will usually accompanied with a dramatically increase of flow resistance. The main objective of present paper is to propose an optimization design theory and develop a corresponding approach to design heat transfer unit, in which the heat transfer and flow resistance will be considered simultaneously. As an example, this paper shows how to obtain a high performance tube in convective heat transfer by optimizing the shape profile using the theory and approach. A 2-D model is used in a direct problem solver, which is solved by finite element software and provides the numerical results in the optimization. Power consumption of flow field P-f is used to measure the flow resistance while generalized thermal resistance R-h is used to measure the thermal resistance. Meanwhile, genetic algorithm (GA) and simplified conjugate-gradient method (SCGM) are applied in this study to optimize the objective function composed by these two aspects. Based on the results of numerical calculation, the optimal velocity field with 61.93% decrease in P-f and 17.13% increase in Rh is obtained. Subsequently, the performances at different inlet velocities V-i are investigated. It is found that the shape profile obtained at V-i = 0.1 m/s always has a better performance than the circular one, even if it is not the ideal shape. Finally, the further optimizations based on different V-i and objective functions are discussed. The results prove that the proposed optimization method is effective in designing the shape of tube. (C) 2015 Elsevier Ltd. All rights reserved.