AlGaN/GaN metal-oxide-semiconductor high electron mobility transistors (MOS-HEMTs) using electron-beam evaporated high-dielectric-constant (high-k) lanthanum oxide layer (La2O3) as the gate insulator have been investigated and compared with the traditional GaN HEMTs. The dielectric constant of the La2O3 insulator layer developed in this study was 13.1. In addition, a negligible hysteresis voltage shift in the capacitance-voltage curves can be obtained after high temperature annealing. The compositions and the crystalline structures of La2O3 with different annealing temperatures were observed by X-ray photoelectron spectroscopy and X-ray diffraction, respectively. The La2O3 thin film achieved a good thermal stability after 200, 400, and 600 degrees C postdeposition annealing owing to its high binding energy (835.7 eV) characteristics. Moreover, the gate leakage current of a traditional metal gate GaN HEMT can be suppressed for 1 order of magnitude after inserting a La2O3 insulator between Ni and AlGaN, resulting in a better pulsed-mode operation. The device linearity was also improved due to its flat and wide transconductance (g(m)) distribution, which was analyzed by a polynomial curve-fitting technique. Therefore, La2O3 is a potential candidate high-k material for the gate insulator to enhance the GaN-based field effect transistor performance while scaling down the device dimension and device reliability at high power operation.