Poly(butyl acrylate)/poly(vinyl acetate-co-methyl methacrylate) PBA/P(VAc-co-MMA) coreshell rubber particles with various shell compositions, i.e., VAc/MMA weight ratios, were used to toughen unsaturated polyester. The morphology and surface-free energy of the rubber particles were determined by transmission electron microscopy (TEM) and contact angle measurements, respectively. The effect of shell structure on the dispersion state of rubber particles inside the unsaturated polyester resin was studied by scanning electron microscopy and TEM. Increasing MMA units in the shell changed the particle dispersion state from small agglomerates or globally well-dispersed particles to large aggregates in the cured-resin matrix. For the blends that contain 5 wt% rubber, the highest un-notched impact toughness, stress-intensity factor (KIC), and fracture energy (GIC) were observed for the blend containing PVAc shell particles. The results showed that by increasing the particle level from 5 to 10 wt%, the highest KIC and GIC values were obtained for the blend containing rubber particles with VAc/MMA (80/20 wt/wt) copolymer shell. The crack-tip damage zone in the neat and rubber-modified unsaturated polyester resins was observed by means of transmission optical microscopy. In addition, using PVAc shell particles exhibited a minimum reduction in the volume shrinkage and tensile properties of the rubber-modified resin. POLYM. ENG. SCI., 52:19281937, 2012. (c) 2012 Society of Plastics Engineers