The initial growth process of Si1-x-yGexCy epitaxial thin films on Si(0 0 1) surfaces is investigated by scanning tunneling microscopy (STM). The effects of the substrate temperature, the film thickness, and the C fraction in Si1-x-yGexCy on the film morphology are systematically examined on an atomic scale. At low temperatures, multi-layered islands are formed and this islanding is closely related to C atoms segregating to the surface during the film growth. As the temperature increases, the growth mode turns into a step-flow process, although high temperature growth promotes C segregation. In the growth of Si1-x-yGexCy films with large C fractions, phase separation between Si-C and Si-Ge, concomitant with C condensation and three-dimensional (3D) islanding on the surface of the growing films, is dominant. We find that the formation of a thin Si1-xGex layer prior to Si1-x-yGexCy film growth drastically improves the film structure, leading to a planar morphology even with large C fractions present in the film. (C) 2003 Elsevier Science B.V. All rights reserved.