This study investigated the mechanism of improving the surface hydrophobicity of low-rank coal by adsorbing dodecyltrimethylammonium bromide (DTAB). Experimental tests were conducted to analyze the adsorption characteristic of DTAB on the coal surface including the adsorption amount, Zeta potential, wetting heat, and X-ray photoelectron spectroscopy measurement. Furthermore, the adsorption configuration, spatial location of the simulation systems, and the coal-water interaction were investigated by molecular dynamics (MD) simulation at the atomic scale. Experimental results indicate that the hydrophobicity of the low-rank coal first increased, then decreased as the DTAB concentration increased. The electrostatic force plays a dominant role in the adsorption of DTAB on low-rank coal surface. The adsorption of DTAB reduced the wetting heat between water and low-rank coal because the oxygen-containing groups were covered. The decrease in hydrophobicity at high DTAB concentration was due to the formation of a bilayer or micelle adsorption. MD simulation results show that the adsorption of DTAB reduced the thickness of the water adsorption layer. The nitrogen atoms of DTAB were oriented toward the coal surface, such that the alkyl chains were oriented to the water phase, thereby producing an additional repulsive effect and subsequently inhibiting the adsorption of water molecules. As a result, the number of hydrogen bonds and the interaction energy between water molecules and low-rank coal decreased, indicating that the hydrophobicity of low-rank coal was enhanced.