The dielectric constant for water is reduced under confinement. Although this phenomenon is well known, the underlying physical mechanism for the reduction is still in debate. In this work, we investigate the effect of the orientation of hydrogen bonds on the dielectric properties of confined water using molecular dynamics simulations. We find a reduced rotational diffusion coefficient for water molecules close to the solid surface. The reduced rotational diffusion arises due to the hindered rotation away from the plane parallel to the channel walls. The suppressed rotation in turn affects the orientational polarization of water, leading to a low value for the dielectric constant at the interface. We attribute the constrained out-of-plane rotation to originate from a higher density of planar hydrogen bonds formed by the interfacial water molecules.