A new macromolecular coupling agent butyl acrylate (BA)-methyl methacrylate (MMA)-vinyl triethoxy silane (VTES) tercopolymer was synthesized using solution polymerization initiated by free radical initiator benzoyl peroxide (BPO) and dicumyl peroxide (DCP). Dodecylthiol is choosed as the chain transfer to control the molecule weight of this tercopolymer. The terpolymer's molecular structure was confirmed by FTIR and NMR, and its average molecular weight was determined by GPC. In this work, the tercopolymer BA-MMA-VTES is used for surface modification of silicon nitride (Si3N4) nanopowder. The structure surface properties and thermal stability of modified nano-Si3N4 were systematically investigated by FTIR, TGA, TEM, and size distribution analyzer. The results show that the macromolecular coupling agent bonds covalently on the surface of nano-sized Si3N4 particles and an organic coating layer is formed. The optimum loading of this macromolecular coupling agent BA-MMA-VTES tercopolymer is 5% (wt %) of nano-sized Si3N4. TEM also reveals that modified nanO-Si3N4 possesses good dispersibility. (C) 2007 Wiley Periodicals, Inc.