Swollen physical magnetorheological (MR) gels were obtained by self-assembling of triblock copolymers containing dispersed soft magnetic particles. The transient rheological responses of these systems were investigated experimentally. Upon sudden application of a homogeneous magnetic field step change, the storage modulus of MR gels continued to increase with time. Such increase trend of the storage modulus could be expressed by a double-exponential function with two distinct modes, a fast and a slow one. The result was compared with the transient rheological response of equivalent MR fluids (paraffin oil without copolymer) and a MR elastomer (PDMS) and interpreted as the consequence of strong rearrangement of the original particle network under magnetic field. Similar to the structure evolution of MR fluids, the ensemble of results suggests that "chaining" and "clustering" processes are also happening inside the gel and are responsible for the rheological behavior, provided they are happening on a smaller length scale (long chains and clusters are hindered). We show that response times of several minutes are typical for the slow response of MR gels. The characteristic time t(2) for the slow process is significantly dependent on the magnetic flux density, the matrix viscoelastic property, particle volume fraction, and sample's initial particle distribution. In order to validate our results, the role of dynamic strain history was clarified. We show that, in the linear viscoelastic region, the particle rearrangement of MR gels was not hindered or accelerated by the dynamic strain history.