Controlling thermal properties is central to many applications, such as thermoelectric energy conversion and the thermal management of integrated circuits. Progress has been made over the past decade by structuring materials at different length scales, but a clear relationship between structure size and thermal properties remains to be established. The main challenge comes from the unknown intrinsic spectral distribution of energy among heat carriers. Here, we experimentally measure this spectral distribution by probing quasi-ballistic transport near nanostructured heaters down to 30 nm using ultrafast optical spectroscopy. Our approach allows us to quantify up to 95% of the total spectral contribution to thermal conductivity from all phonon modes. The measurement agrees well with multiscale and first-principles-based simulations. We further demonstrate the direct construction of mean free path distributions. Our results provide a new fundamental understanding of thermal transport and will enable materials design in a rational way to achieve high performance.