Ternary composites of high-impact polystyrene (HIPS), elastomer, and magnesium hydroxide filler encapsulated by polystyrene were prepared to study the relationships between their structure and mechanical properties. Two kinds of morphology were formed. Separation of elastomer and filler was found when a nonpolar poly[styrene-b(ethylene-co-butylene)-b-styrene] triblock copolymer (SEBS) was incorporated. Encapsulation of filler by elastomer was achieved by using the corresponding maleinated SEBS (SEBS-g-MA). The mechanical properties of ternary composites were strongly dependent on microstructure. In this study, the composites with separate dispersion structure showed higher elongation, modulus and impact strength than those of encapsulation structure. Impact-fracture surface observation showed that the toughening mechanism was mainly due to the massive cavitation and extensive matrix yielding. (c) 2006 Wiley Periodicals, Inc.