Inorganic nanoparticles are an attractive material that shows unique properties that differ from their bulk counterparts. Assembly of nanoparticles with soft materials is an effective approach to leverage their unusual properties for the fabrication of functional devices. Among the various soft materials, polymer brushes are expected to offer exciting opportunities due to their unique conformational properties. Here, we review research progress in the assembly and active control of gold nanoparticles with polymer brushes as a scaffold.