The combination of its high electron mobility(1-5), broadband absorption(6) and ultrafast luminescence(7-10) make graphene attractive for optoelectronic and photonic applications(11-13), including transparent electrodes(14), mode-locked lasers(15) and high-speed optical modulators(16). Photo-excited carriers that have not cooled to the temperature of the graphene lattice are known as hot carriers, and may limit device speed and energy efficiency. However, their roles in charge and energy transport are not fully understood(17-20). Here, we use time-resolved scanning photocurrent microscopy to demonstrate that hot carriers, rather than phonons, dominate energy transport across a tunable graphene p-n junction excited by ultrafast laser pulses. The photocurrent response time varies from 1.5 ps at room temperature to 4 ps at 20 K, implying a fundamental bandwidth of similar to 500 GHz (refs 12,13,21). Gate-dependent pump-probe measurements demonstrate that both thermoelectric and built-in electric field effects contribute to the photocurrent, with the contribution from each depending on the junction configuration. The photocurrent produced by a single pulsed laser also displays multiple polarity reversals as a function of carrier density, which is a possible signature of impact ionization(19,22,23).