The influence of nanomaterials such as Multi-Walled Carbon NanoTubes (MWNT) and organoclay (Cloisite 30B) on the physical and mechanical properties of thermoset matrix such as Unsaturated Polyester (UP) resins is investigated. Although styrene containing UP resins have a wide spread application in industry, lack of information exists regarding the behavior of MWNT/organoclay/polyester ternary nanocomposite systems. The main aim of this research was first to evaluate the effect of nanofiller on the flammability of UP resins and, second, to characterize their mechanical properties such as toughness and their tensile strength. The rheological studies showed shear thinning for samples of UP resins containing MWNT and Cloisite 30B. The cone calorimetry measurement was used to evaluate the flame-retardency, the gas emission of the nanocomposite and whether or not this system can be designated as a nanocomposite. This was understood in the test by the peak heat release rate being lowered and shifted to shorter times. Furthermore, the tensile and impact properties of samples were evaluated. The obtained results indicated that nanofiller particles caused both increase and decrease in the impact and tensile strength. (C) 2011 Wiley Periodicals, Inc. J Appl Polym Sci, 2012