Recent work on layered perovskites has established the group theoretical guidelines under which a combination of octahedral distortions and cation ordering can break inversion symmetry, leading to polar structures. The microscopic mechanism of this form of ferroelectricity-so-called hybrid-improper ferroelectricity-has been elucidated in two families of layered perovskites: AA'B2O6 double perovskites and Ruddlesden-Popper phases. In this work, we use symmetry principles, crystal chemical models, and first-principles calculations to unravel the crystal chemical origin of ferroelectricity in the Dion Jacobson phases, and show that the hybrid improper mechanism can provide a unifying explanation for the emergence of polar structures in this family of materials. We link trends in the magnitude of the induced polarizations to changes in structure and composition and discuss possible phase-transition scenarios. Our results suggest that the structures of several Dion Jacobson phases that have previously been characterized as centrosymmetric should be re-examined. Our work adds new richness to theories of how polar structures emerge in layered perovskites.