Organic-inorganic hybrid materials, such as polyhedral oligomeric silsesquioxanes (POSS), have the potential to improve the mechanical properties of methacrylate based composites and resins used in dentistry. In this article, the authors investigated the effect of the concentration of trismethacryl isobutyl POSS on the in vitro friction and wear of a composite resin mainly composed of ethoxylated bisphenol A dimethacrylate and triethyleneglycol dimethacrylate and filled with silanated silica glass. Experiments on both de-ionized water and saliva substitute were conducted. The effect of POSS on the friction and wear of the unfilled polymer resin immersed in saliva substitute were also investigated. From pin-on-disk experiments, it was found that the addition of small amounts (0.5-3 wt %) of POSS improves the friction and wear resistance of both the resin composite and the polymer resin. Samples with a higher POSS concentration (5%) showed an opposite effect, which illustrates the limited range where the reinforcement properties of POSS can operate. This concentration dependence is attributed to phase separation at higher concentrations of POSS, which affects the structural integrity, and hence, the friction and wear properties of the resin composite and the polymer resin. Depth dependence of the coefficient of friction was observed in the composite, indicating inhomogeneity from curing. The results presented in this article can serve as a useful guide in the development of novel reinforced resin composites and polymer resins for dental applications. (C) 2010 American Vacuum Society. [DOI: 10.1116/1.3299103]