Curing kinetics and mechanism determine the structure and property of thermosetting resins and related composites. The curing kinetics and mechanism of a novel high performance resin system based on hyper-branched polysiloxane (HBPSi), 2,2'-diallylbisphenol A modified bismaleimide (BD), and cyanate ester (CE) resins for Resin Transfer Molding (RTM) technique were systemically studied by Differential Scanning Calorimetry (DSC), Fourier Transform Infrared (FTIR) spectra, and torque rheometer. Results show that the addition of HBPSi to BD/CE resin not only decreases the initial curing temperature and apparent activation energy, but also changes the curing mechanism, and thus the structure and properties of resultant crosslinked networks. An "Interpenetrating network (IPN)-coupling structure" is proposed to be formed in the HBPSi/BD/CE system, which is different from traditional "IPN" structure in BD/CE resin. The simulation of curing reaction suggests that the variety of the curing activity leads to the difference between the curing behaviors of BD/CE and HBPSi/BD/CE resins, which is in good agreement with FTIR and DSC analyses. (c) 2011 Wiley Periodicals, Inc. J Appl Polym Sci 122: 304-312, 2011