Device quality GaAs epilayers were grown by molecular beam epitaxy using Si-on-insulator (SOI) as a compliant substrate. The density of dislocations is significantly reduced in the GaAs epilayers. Based on the dislocation theory and detailed experiments, a model of compliant growth to reduce dislocation density has been developed. As compared to the conventional growth technique, the advantage of reduction of dislocation density in GaAs epilayers grown on a much thinner compliant Si membrane can be predicted with this model. Our experimental results indicate that compliant growth can improve the crystal quality of epilayers on lattice-mismatched substrates significantly. Utilizing this growth technique, the full width at half-maximum of the x-ray rocking curves is measured to be as low as 128 arcsec for 3-mum-thick GaAs epilayers grown on SOI compliant substrates. Furthermore, structures of AlGaAs/GaAs heterojunction bipolar transistor (HBT) have been grown by compliant growth on SOI substrates. The corresponding I-V characteristics demonstrate good performance of the HBTs.