Epoxy matrix nanocomposites were prepared by diglycidyl ether of bisphenol A (DGEBA) modified with epoxide polyurethane (EPU) containing oxazone-group and reinforced by low concentrations of octaaminophenyl polyhedral oligomeric silsesquioxane (OAPOSS). The thermal, mechanical, and thermomechanical properties of as-prepared hybrid nanocomposites were studied. The results demonstrate that the thermal stability, tensile strength, glass transition temperature, and storage modulus of epoxy polymer/EPU blends are all significantly improved with OAPOSS as an reinforcement. The inclusion of EPU in the epoxy polymer helps improve the toughness of the matrix, providing higher impact strength, and higher elongation at break. Tribological testing shows that the wear rate of the hybrid nanocomposite is reduced by more than 80% at 0.1 wt% OAPOSS compared with pure epoxy-EPU, while the friction coefficient is nearly unchanged. SEM-EDS analysis of worn scar indicates that the wear mechanism of OAPOSS incorporated hybrid composites is the coexistence of abrasive wear and adhesive wear. The improved properties are due to the uniform dispersion of OAPOSS in the epoxy-EPU matrix and network formation by cross-linking and chemical bonds from OAPOSS reacted with epoxy-EPU matrix.