We report how to control the self-assembly of magnetic nanoparticles and a prototypical amphiphilic block-copolymer composed of poly(acrylic acid) and polystyrene (PAA-b-PS). Three distinct structures were obtained by controlling the solvent-nanoparticle and polymer-nanoparticle interactions: (1) polymersomes densely packed with nanoparticles (magneto-polymersomes), (2) core-shell type polymer assemblies where nanoparticles are radially arranged at the interface between the polymer core and the shell (magneto-core shell), and (3) polymer micelles where nanoparticles are homogeneously incorporated (magneto-micelles). Importantly, we show that the incorporation of nanoparticles drastically affects the self-assembly structure of block-copolymers by modifying the relative volume ratio between the hydrophobic block and the hydrophilic block. As a consequence, the self-assembly of micelle-forming block-copolymers typically produces magneto-polymersomes instead of magneto-micelles. On the other hand, vesicle-forming polymers tend to form magneto-micelles due to the solubilization of nanoparticles in polymer assemblies. The nanoparticle-polymer interaction also controls the nanoparticle arrangement in the polymer matrix. In N,N-dimethylformamide (DMF) where PS is not well-solvated, nanoparticles segregate from PS and form unique radial assemblies. In tetrahydrofuran (THF), which is a good solvent for both nanoparticles and PS, nanoparticles are homogeneously distributed in the polymer matrix. Furthermore, we demonstrated that the morphology of nanoparticle-encapsulating polymer assemblies significantly affects their magnetic relaxation properties, emphasizing the importance of the self-assembly structure and nanoparticle arrangement as well as the size of the assemblies.