The formation mechanism of insulating titanium oxynitride nanolayers was studied by means of spectroscopic ellipsometry. The parameters of the model for solving the inverse problem of ellipsometry were chosen on the basis of experimental data obtained with the help of high-resolution transmission electron microscopy, atomic force microscopy, UV and X-ray photoemission spectroscopy. The layers were obtained using the plasmachemical nitridation of thin titanium layers on silicon substrate. The features of nanolayer preparation procedure (low temperature and short process time), as well as good masking characteristics (the minimal density of pores and defects, and the perfect smoothness of the surface) allow one to use these layers for chemical and electron passivation and stabilization of the surface of semiconductor nano-objects (quantum dots, quantum wires, nanowhiskers etc.) for electron and photon nanodevices. (C) 2011 Elsevier B.V. All rights reserved.