Hydroxyl terminated poly(ether ether ketone) oligomer with pendant methyl group (PEEKMOH) was prepared. Ternary nanocomposites were processed by blending PEEKMOH oligomer with diglycidyl ether of bisphenol-A (DGEBA) epoxy resin along with organically modified montmorillonite (Cloisite 25A) followed by curing with 4,4'-diamino diphenyl sulfone. Tensile moduli and flexural moduli were increased, while the tensile strength and Izod impact strength were decreased with increase in clay content. Similarly, storage moduli and loss moduli were increased and glass transition temperature was decreased as the percentage of clay increased. X-ray diffractograms showed exfoliated morphology even with higher concentration of clay content (8 phr). Scanning electron microscopy of fractured surfaces and tensile failed specimens revealed slow crack propagation and increase in river markings with nanoclay incorporation confirming the improvement in toughness. The domain size of PEEKMOH was decreased with the incorporation of nanoclay into the epoxy matrix, indicating the restriction of growth mechanism by nucleation during phase separation. With increase in clay content, phase separation disappeared indicating gelation occurs before phase separation. Fracture toughness was increased with the addition of PEEKMOH and clay in epoxy resin. Coefficient of thermal expansion of nanocomposites decreases up to 3 phr clay concentrations thereafter it increases. A marginal increase in thermal stability was observed with increase in clay content. (c) 2007 Wiley Periodicals, Inc.