We investigated the growth of the epitaxial Ge layers in a 40 nm wide SiO2 trench array on Si by ultra-high vacuum chemical vapor deposition. If the thickness of Ge was less than the height of the SiO2 trenches, the Ge layers grew epitaxially by a selective epitaxial growth process without any detectable surface modification, which is due to the high interfacial energy between the SiO2 mask and Ge. We calculated the critical strain required to modify the Ge surface via 3-dimensional island transition (the minimum strain) as a function of the trench width. Considering the energies involved in the transition, we found that uniformly shaped Ge layers along the trenches were energetically more favorable than those with surface undulations as the width of the trench decreased. The strained Ge epilayers relaxed their energy by forming the defects, such as dislocations at the Ge/Si interfaces and stacking faults. From the strain analyses, the residual strains for parallel and perpendicular to the trench direction in the Ge layers were -0.72% and -0.22%, respectively. (C) 2013 Elsevier B.V. All rights reserved.