The ability of energy carriers to move between atoms and molecules underlies biochemical and material function. Understanding and controlling energy flow, however, requires observing it on ultrasmall and ultrafast spatio-temporal scales, where energetic and structural roadblocks dictate the fate of energy carriers. Here, we developed a non-invasive optical scheme that leverages non-resonant interferometric scattering to track tiny changes in material polarizability created by energy carriers. We thus map evolving energy carrier distributions in four dimensions of spacetime with few-nanometre lateral precision and directly correlate them with material morphology. We visualize exciton, charge and heat transport in polyacene, silicon and perovskite semiconductors and elucidate how disorder affects energy flow in three dimensions. For example, we show that morphological boundaries in polycrystalline metal halide perovskites possess lateral- and depth-dependent resistivities, blocking lateral transport for surface but not bulk carriers. We also reveal strategies for interpreting energy transport in disordered environments that will direct the design of defect-tolerant materials for the semiconductor industry of tomorrow.