In situ mass spectroscopy of recoiled ions (MSRI) and reflection high-energy electron diffraction (RHEED) are used to monitor the surface structure, and the near surface layer composition during the pre-growth preparation step and the deposition of GaN thin films on Si (001), Al2O3 (0001), and GaAs (001) substrates by electron-cyclotron-resonance (ECR)-assisted molecular beam epitaxy. Due to the high lattice mismatch between GaN thin films and these substrates, the initial nucleation step is found to be critical on the resulting thin film properties. Nitridation of the clean Al2O3 (0001) and GaAs(001) surface is observed by RHEED, and the formation of an AlN/GaN overlayer is confined by MSRI. During GaN deposition on Si, Al2O3, and GaAs, MSRI and RHEED are used to follow the growth mode and coverage. If a RHEED pattern from the nucleating GaN overlayer has a cubic or wurtzitic symmetry and the MSRI spectra-which is sensitive to the first few monolayers-shows the presence of Ga and N, only then the growth is layer by layer. Scanning electron microscopy micrographs also show a smooth and continuous surface. If an amorphous RHEED pattern is observed and the presence of the Si (Al, O in the case of Al2O3 and As in the case of GaAs) signal is detected in addition to Ga and N, the growth is columnar. In conclusion, we show that RHEED and MSRI form a powerful combination of thin film surface characterization tools ideally suited for real-time optimization of substrate preparation and thin film growth parameters for the nitride materials system.