Formation of benzene excimer following UV excitation of the neat liquid is monitored with femtosecond spectroscopy. A prompt rise component in excimer transient absorption, which contradicts the classical scenario of gradual reorientation and pairing of the excited monomers, is observed. Three-pulse experiments in which the population of evolving excimers is depleted by a secondary dump pulse demonstrate that the excimer absorption band is polarized along the interfragment axis. The experiments furthermore prove that the subsequent 4-fold increase in excimer absorption over similar to 50 ps is primarily due to an increase in the transition dipole of pairs which are formed early on, and not to excited monomers forming excimers in a delayed fashion due to unfavorable initial geometry. Results are analyzed in light of recent studies of local structure in the liquid benzene combined with advanced electronic structure calculations. The prompt absorption rise is ascribed to excited states delocalized over nearby benzene molecules, which are sufficiently close and nearly parallel in the pure liquid. Such low-symmetry structures, which differ considerably from the optimized structures of isolated benzene dimer and solid benzene, are sufficiently abundant in liquid benzene. Electronic structure calculations confirm the orientation of transition dipoles of the excimers along the interparticle axis and demonstrate how slow refinement of the intermolecular geometry leads to a significant increase in the excimer absorption strength.