A photoresponsive amphiphilic gold nanoparticle (AuNP) is achieved through the decoration of AuNP with hydrophilic poly(ethylene glycol) (PEG) and hydrophobic photoresponsive polymethacrylate containing spiropyran units (PSPMA). Owing to the photoresponsive property of spiropyran units, the amphiphilic AuNPs can easily achieve the controllable assembly/disassembly behaviors under the trigger by light. Under visible light, spiropyran units provide weak intermolecular interactions between neighbored AuNPs, leading to isolated AuNPs in the solution. While under UV light irradiation, spiropyran units in the polymer brushes transform into merocyanine isomer with conjugated structure and zwitterionic state, promoting the integration of adjacent AuNPs through pi-pi stacking and electrostatic attractions, further leading to the formation of Au oligomers. The smart reversible AuNP oligomers exhibited switchable plasmonic coupling for tuning surface-enhanced Raman scattering (SERS) activity, which is promising for the application of SERS based sensors and optical imaging.