We investigated the surface morphology of 50-nm-thick GaP layers grown on Si substrates, by atomic force microscopy. Pits with a density of about 10(8) cm(-2) were observed at the surface of GaP layers grown by conventional migration-enhanced epitaxy (MEE). The pit diameter increased with increasing the growth temperature in the range from 440 degrees C up to 540 degrees C. Transmission electron microscope (TEM) observations revealed that the origin of the pits was the melt-back etching of the Si surface by Ga droplets. The Ga droplets are thought to form during the initial MEE growth on a P-prelayer. By changing the conventional MEE growth mode into molecular beam epitaxy growth mode for the initial 2 monolayers, the melt-back etching was suppressed. As a result, a pit-free GaP layer was successfully grown on a Si substrate with a root-mean-square surface roughness of 0.2 nm. in addition, no stacking faults were observed in cross-sectional TEM images. Finally, anti-phase domains completely self-annihilated within 10 nm from the interface. (C) 2008 Elsevier B.V. All rights reserved.