Polystyrene and poly(methyl methacrylate/styrene) were prepared by in situ polymerization in the presence of Cloisite 20A, and were then used to synthesize unsaturated polyester-based thermoplastic/thermoset hybrids with improved properties. This approach allows for an increased degree of dispersion and delamination of the silicate layers as well as the presence of thermoplastic chains within the thermoset resin, which improve the physical properties of the system. During curing, methyl methacrylate promotes the conversion of styrene inside the clay galleries and also in the thermoplastic-rich phase. X-ray diffraction and transmission electron microscopy (TEM) revealed a fine intercalated/exfoliated structure in the nanocomposites. Fracture tests showed that a combination of clay and thermoplastic resulted in a synergistic improvement of the fracture toughness of the nanocomposite while stiffness was maintained at the level of the unmodified polyester. The hybrid systems exhibited spherical domains of thermoplastic-rich phase dispersed in the thermoset matrix. The clay was present only in the thermoplastic-rich phase with a portion congregated at the interface between the thermoset and thermoplastic domains. Interestingly, morphological studies showed that the clay layers surrounded microgels of thermoset contained within the thermoplastic domains. Such microgels of thermoset are thought to be fewer and smaller in the hybrid systems without clay, leading to smaller and more compliant dispersed phase domains at the same thermoplastic content and reduced fracture toughness. (C) 2012 Elsevier Ltd. All rights reserved.