The blinking of surface-enhanced Raman scattering (SERS) signal was observed for Ag touching particles with an extremely small number of molecules progressively extinguished in 10-30 min probably because of thermal diffusion to marginal sites from the junction or complete desorption of adsorbates. Interestingly, we found a significant change in elastic scattering spectra observed with a white light source during the inactivation process of SERS. Furthermore, it turned out exploiting numerical simulations for the local electric field and scattering spectra that such spectral changes result from the distinct electromagnetic coupling between the localized surface plasmon (LSP) and absorption of dye according to the on/off features of SERS activity. This is induced by thermal diffusion of individual molecules in and out of the junctions in accordance with the experimental observations on the vast enhancement in SERS. Thus, the critical importance of the junctions of Ag particles was confirmed concerning the origin of the blinking and mechanism of the enormous enhancement in SERS.