The effect of rubber nanoparticles on mechanical properties and fracture toughness was investigated. Rubber nanoparticles of 2-3 nm were in situ synthesized in epoxy taking advantage of the reaction of an oligomer diamine with epoxy. The chemical reaction was verified by gel permeation chromatography (GPC) and (HNMR)-H-1, and the microstructure was characterized by transmission electron microscope. The rubber nanoparticles caused much less Young's modulus deterioration but toughened epoxy to a similar degree in comparison with their peer liquid rubber that formed microscale particles during curing. Fifteen wt % of rubber nanoparticles increased fracture energy from 140 to 840 J/m(2) with Young's modulus loss from 2.85 to 2.49 GPa. The toughening mechanism might be the stress relaxation of the matrix epoxy leading to larger plastic work absorbed at the crack Lip; there is no particle cavitation or deformation.: neither crack deflection nor particle bridging were observed. The compound containing rubber nanoparticles demonstrates Newtonian liquid behavior with increasing shear rate; it shows lower initial viscosity at low shear rate than neat epoxy; this provides supplementary evidence to NMR and GPC result. (C) 2008 Wiley Periodicals, Inc.