Recent measurements on cathodoluminescences spectra of natural and isotope-substituted boron nitride nanotubes (BNNTs) surprisingly suggest the existence of pronounced radiative transitions in BN tubes, which are possibly induced by the oxygen substitutional impurities of the samples. [Han, W. Q. et al. Nano Lett. 2008, 8, 491] However, the structural pattern of the O-doped BN tube is unknown, as a result, how does the substitutional impurity in BNNT contribute to the observed radiative transitions is still a puzzle. Using first-principle calculations, we predict a novel, stable O-doped BNNT configuration. Such a structure contains one B3O6 group, which is similar to the structural unit of boron oxide. Our calculations demonstrate that this type of O substitutional impurity can result in some donor-like gap states in the electronic structure and lead to the significant changes on the optical properties of BNNTs. The vibrational properties of the O-doped BNNT and boron oxide are also investigated. Our work elucidates the origins for experimental findings and provides a strong theoretical evidence on the existence of O substitutional impurity-induced radiative transitions in BNNT systems.