Selective epitaxial growth of Ge and Ge1-xSnx layers on Si substrates was performed by using metal-organic chemical vapor deposition (MOCVD) with precursors of tertiary-butyl-germane (t-BGe) and tri-butyl-vinyl-tin (TBVSn). We investigated the effects of growth temperature and total pressure during growth on the selectivity and the crystallinity of the Ge and Ge1-xSnx epitaxial layers. Under low total pressure growth conditions, the dominant mechanism of the selective growth of Ge epitaxial layers is the desorption of the Ge precursors. At a high total pressure case, it is needed to control the surface migration of precursors to realize the selectivity because the desorption of Ge precursors was suppressed. The selectivity of Ge growth was improved by diffusion of the Ge precursors on the SiO2 surfaces when patterned substrates were used at a high total pressure. The selective epitaxial growth of Ge1-xSnx layer was also realized using MOCVD. We found that the Sn precursors less likely to desorb from the SiO2 surfaces than the Ge precursors.