The homogeneous Yb3+/Al3+/B3+-co-doped silica glasses were prepared via a sol-gel method. The impact of B2O3 addition on the physical and optical properties and network structure was systematically studied. The network structure was investigated by the Fourier Transform Infrared (FT-IR), Raman spectra, and Solid State Nuclear Magnetic Resonance (SSNMR). Herein, B2O3 addition can continuously decrease the refractive index and density. When B2O3 is lower than 2 mol%, B2O3 addition can obviously decrease the scalar crystal field parameters, Yb3+ asymmetry degree, Yb3+ cross-sections, due to the generation of Yb-O-B bonds at the cost of partial Yb-O-Al/Si ones. When B2O3 is more than 2 mol%, FT-IR, Raman spectra, and SSNMR results indicate that further increased B atoms prefer to connect with Si and Al rather than Yb. Consequently, the above parameters are basically unchanged. Based on the results, an intuitive model of structure and properties evolution during the substitution of SiO2 by B2O3 has been established.