Here we study experimentally and by simulations the interaction of monovalent organic and inorganic anions with hydrophobic and hydrophilic colloids. In the case of hydrophobic colloids, our experiments show that charge inversion is induced by chaotropic inorganic monovalent ions but it is not induced by kosmotropic inorganic anions. For organic anions, giant charge inversion is observed at very low electrolyte concentrations. In addition, charge inversion disappears for both organic and inorganic ions when turning to hydrophilic colloids. These results provide an experimental evidence for the hydrophobic effect as the driving force for both ion specific effects and charge inversion. In the case of organic anions, our molecular dynamics (MD) simulations with full atomic detail show explicitly how the large adsorption free energies found for hydrophobic colloids are transformed into large repulsive barriers for hydrophilic colloids. Simulations confirm that solvation free energy (and hence the hydrophobic effect) is responsible for the build up of a Stern layer of adsorbed ions and charge inversion in hydrophobic colloids and it is also the mechanism preventing charge inversion in hydrophilic colloids. Overall, our experimental and simulation results suggest that the interaction of monovalent ions with interfaces is dominated by solvation thermodynamics, that is, the chaotropic/kosmotropic character of ions and the hydrophobic/hydrophilic character of surfaces.