Gold nanoparticles and sub-nanoparticles famously act as highly efficient and selective low-temperature oxidation catalysts with molecular oxygen, in stark contrast to the nobility of the bulk phase. The origins of this activity and the nature of the active species remain open questions. Gas-phase studies of isolated gold clusters hold promise for disentangling these problems. Here we address the interaction of neutral gold clusters (Au-n; 4 <= n <= 21) with molecular oxygen by probing the highly characteristic O-O vibrational stretch frequencies. This reveals that for selected cluster sizes the oxygen is highly activated with respect to the free moiety. Complementary quantum chemical calculations provide evidence for substantial electron transfer to the O-2 unit and concomitant rearrangement of the parent gold cluster structure upon binding and activation. This gives evidence for a model of the interaction between neutral gold clusters and molecular oxygen.