We report the synthesis of colloidal CsPbX3-Pb4S3Br2 (X = Cl, Br, I) nanocrystal heterostructures, providing an example of a sharp and atomically resolved epitaxial interface between a metal halide perovskite and a non-perovskite lattice. The CsPbBr3-Pb4S3Br2 nanocrystals are prepared by a two-step direct synthesis using preformed subnanometer CsPbBr3 clusters. Density functional theory calculations indicate the creation of a quasi-type II alignment at the heterointerface as well as the formation of localized trap states, promoting ultrafast separation of photogenerated excitons and carrier trapping, as confirmed by spectroscopic experiments. Postsynthesis reaction with either Cl- or I- ions delivers the corresponding CsPbCI3-Pb4S3Br2 and CsPbI3-Pb4S3Br2 heterostructures, thus enabling anion exchange only in the perovskite domain. An increased structural rigidity is conferred to the perovskite lattice when it is interfaced with the chalcohalide lattice. This is attested by the improved stability of the metastable gamma phase (or "black" phase) of CsPbI3 in the CsPbI3-Pb4S3Br2 heterostructure.