Organic-inorganic hybrids involving cyanate ester and hydroxyl-terminated polyclimethylsiloxane (HTPDMS) modified diglycidyl ether of bisphenol A (DGEBA; epoxy resin) filled with organomodified clay [montmorillonite (MMT)] nanocomposites were prepared via in situ polymerization and compared with unfilled-clay macrocomposites. The epoxy-organomodified MMT clay nanocomposites were prepared by the homogeneous dispersion of various percentages (1-5%), and the resulting homogeneous epoxy/clay hybrids were modified with 10% HTPDMS and gamma-aminopropyltriethoxysilane as a coupling agent in the presence of a tin catalyst. The siliconized epoxy/clay prepolymer was further modified separately with 10% of three different types of cyanate esters, namely, 4,4'dicyanato-2,2'-diphenylpropane, 1,1'-bis(3-methyl-4-cyana-tophenyl) cyclohexane, and 1,3-dicyanato benzene, and cured with diaminodiphenylmethane as a curing agent. The reactions during the curing process between the epoxy, siloxane, and cyanate were confirmed by Fourier transform infrared analysis. The results of dynamic mechanical analysis showed that the glass-transition temperatures of the clay-filled hybrid epoxy systems were lower than that of neat epoxy. The data obtained from mechanical studies implied that there was a significant improvement in the strength and modulus by the nanoscale reinforcement of organomodified MMT clay with the matrix resin. The morphologies of the siloxane-containing, hybrid epoxy/clay systems showed heterogeneous character due to the partial incompatibility of HTPDMS. The exfoliation of the organoclay was ascertained from X-ray diffraction patterns. The increase in the percentage of organomodified MMT clay Lip to 5 wt % led to a significant improvement in the mechanical properties and an insignificant decrease in the glass-transition temperature versus the unfilled-clay systems. (C) 2007 Wiley Periodicals, Inc.