Fe3O4-based magnetic rubbers are desirable because they take advantage of high elasticity, large elongations, and high magnetic saturation intensity of each component. However, direct compounding of rubbers and Fe3O4 usually resulted in low mechanical properties which made the magnetic rubbers unable to satisfy practical applications. In this paper, we prepared nitrile butadiene rubber (NBR)/Fe3O4-based magnetic rubbers with good mechanical properties by means of in situ compatibilization using zinc dimethacrylate (ZDMA). Our strategy was based on the polymerization of ZDMA during a peroxide-induced vulcanization, in which the polymerized ZDMA (PZDMA) brought massive Zn2+ ion pairs into NBR Zn2+ ion pairs not only increased the polarity of NBR but also strongly interacted, with Fe3O4 nanoparticles. Because of the graft-PZDMA, the Zn2+ ion pairs finally strengthened interactions between Fe3O4 nanoparticles and NBR molecules. We hope this report laid an applied foundation for design of high performance magnetic rubbers to satisfy the various potential applications.