Many types of structure of the SK mode have possibility to appear as the crystal grows. In order to understand the most stable structure for the SK mode and the minimum-energy path through which the SK mode structure with the smallest free energy appears at each layer-thickness, the strain, surface, and interfacial energies of the GaPSb/GaP system were calculated for the FM, VW and several types of SK mode structures. The free energy for each structure was derived from these energies. By comparison of the free energies of the several types of the SK structure, it is found that the most stable structure of the SK mode is a cluster with four lattice layers or minimum thickness on a wetting layer whose lattice layer increases. As the crystal grows in the SK mode, the path on which only the wetting layer becomes thicker is the most stable. The path on which only the cluster on the wetting layer becomes larger is the most unstable. The thickness-composition phase diagrams for the FM, SKj (j = 1-2) and VW modes were determined for the GaPSb/GaP(1 1 1) system. The area of the SKj region becomes smaller as the cluster height j increases, and that of the VW region becomes larger. In order to compare these phase diagrams, the phase diagram for the case of the SK mode structure whose wetting layer is always one lattice layer thick was determined, and the phase diagram for the SKI mode was also determined by considering the reconstruction of dangling bonds for the calculation of the surface energy.