The role of low binding energy sites on the adsorption of H-2 in metal organic frameworks (MOFs) with unsaturated metal centers has not been identified. For instance, the importance of the benzene sites on H-2 adsorption at the metal site in MOF-74 has not been established. We report here experimental evidence that unambiguously shows that the internal mode of H-2 adsorbed at the metal site undergoes both a frequency shift and a marked change in its dynamic dipole moment when H-2 is adsorbed at the next nearest neighbor "benzene" site in MOF-74-Co. The effect of loading (i.e., occupation of all benzene sites) also induces spectroscopic shifts in H-2 at the metal site. These interactions highlight the role of lower binding energy sites in H-2 adsorption.