Using a two-step modeling approach, we address the full spectrum of direct, reversed, and altered ionic sequences as the charge of the ion, the charge of the surface, and the surface polarity are varied. From solvent-explicit molecular dynamics simulations, we extract single-ion surface interaction potentials for halide and alkali ions at hydrophilic and hydrophobic surfaces. These are used within Poisson-Boltzmann theory to calculate ion density and electrostatic potential distributions at mixed polar/unpolar surfaces for varying surface charge. The resulting interfacial tension increments agree quantitatively with experimental data and capture the Hofmeister series, especially the anomaly of lithium, which is difficult to obtain using continuum theory. Phase diagrams that feature different Hofmeister series as a function of surface charge, salt concentration, and surface polarity are constructed from the long-range force between two surfaces interacting across electrolyte solutions. Large anions such as iodide have a high hydrophobic surface affinity and increase the effective charge magnitude on negatively charged unpolar surfaces. Large cations such as cesium also have a large hydrophobic surface affinity and thereby compensate an external negative charge surface charge most efficiently, which explains the well-known asymmetry between cations and anions. On the hydrophilic surface, the size-dependence of the ion surface affinity is reversed, explaining the Hofmeister series reversal when comparing hydrophobic with hydrophilic surfaces.