GeSn layers are formed on Si substrates by the sputter epitaxy method. Flat GeSn layers with 11.5 % Sn content are obtained. In the X-ray diffraction spectra of the GeSn layers with 11.5 % Sn content grown at 523 K, a sharp Ge (004) peak appears at about 64.5306 degrees and the full width at half maximum is about 0.0984 degrees. However, the surface segregation of Sn is observed in the GeSn layers with 11.5 % Sn content at growth temperatures of 548 K and above in this study, which increases the surface roughness. The crystallinity of the GeSn layers formed on Si substrates strongly depends on the growth temperature, but the primary factor degrading the crystallinity is the large lattice mismatch between Si and GeSn. According to a transmission electron microscope image, some dislocations appear at the interface of the GeSn layer grown at 523 K and Si owing to the large lattice mismatch, but there is no Sn surface segregation and a highly ordered atomic arrangement is observed in the upper region. It is considered that a high deposition rate limits Sn surface segregation and enables the growth of GeSn layers at relatively high temperatures, resulting in improved crystallinity. The band gap of the GeSn layers with 8.4 % Sn content is determined by Fourier transform infrared spectroscopy measurement and is about 0.52 eV, which indicates that band gap narrowing occurs.