The surface of silicon nanocrystals embedded in an oxide matrix can contain numerous interface defects. These defects strongly affect the nanocrystals photoluminescence efficiency and optical absorption. Dangling-bond defects are nearly eliminated by H2 passivation, thus decreasing absorption below the quantum-confined bandgap and enhancing PL efficiency by an order of magnitude. However, there remain numerous other defects seen in absorption by photothermal deflection spectroscopy; these defects cause non-radiative recombination that limits the PL efficiency to <15%. Using atomistic pseudopotential simulations, we attribute these defects to two specific types of distorted bonds: Si-Si and bridging Si-O-Si bonds between two Si atoms at the nanocrystal surface.