The formation process of Ti1-xNbxO2 films with the Nb concentration up to 38 at.% by metal-organic chemical vapor deposition (MOCVD) using titanium tetraisopropoxide (TTIP) and pentaethoxy niobium (PEN) as precursors has been characterized by In-situ Fourier transform IR (FTIR) spectroscopy and spectroscopic ellipsometry. Ln-situ FTIR data suggested that there was no gas phase decomposition of TTIP near the substrate surface under typical growth conditions. Post-deposition analysis of In-situ ellipsometric data allowed simultaneous determination of film thickness, growth rate, and refractive index of the films. Growth mode and surface roughness were also obtained in the post-deposition analysis. The growth process of Ti1-xNbxO2 films can be divided into three stages: nucleation and coalescence, the homogeneous growth, and the surface roughness evolution. Increase of the growth temperature appears to increase the initial island size, and decrease the outer rough surface layer. X-ray photoelectron spectroscopy revealed that the oxidation state of Ti and Nb is 4+. The structure of the Ti1-xNbxO2 films strongly depends on the growth temperature but not on the Nb concentration. The Ti1-xNbxO2 films deposited at 400 degrees C are pure anatase, a;nd at 800 degrees C are pure rutile, but show a mixture of anatase and rutile at the temperature range from 550 degrees C to 800 degrees C. Such a phase change in the films results in a large range of refractive index for the Ti1-xNbxO2 films.