The high viscosity of ionic liquids (ILs) is one of the impediments to their application in industry. Understanding the relationship between structure and viscosity is a key issue for the directed design of ILs with low viscosity. In this work, the microstructures and interactions of three representative imidazolium-based ILs were studied by quantum chemistry calculations and molecular dynamics simulations to investigate the origin of different viscosities. An all-atom force field for difluorophosphate ([PO2F2]) anion was developed. The sandwich structures of hydrogen-bond networks were observed. A relationship between the number and energy of hydrogen bonds and the viscosity was proposed. The order of interaction energies is consistent with the trend of experimental viscosities. The simulation studies suggest that the hydrogen bonds and interaction energy play important roles in determining the viscosity of an IL.