Varying systematically the structure of glassy poly(amide imide), poly(ester imide), and polyimide, we have studied the correlation between free volume and transport properties of highly selective polymer membranes. Free volume data were determined by means of positron annihilation lifetime spectroscopy (PALS) while transport properties originate from time-lag measurements of permanent gases. We find a good correlation between PALS average hole size and transport coefficients. The correlation is much better than with free volume data from group contribution methods. It is shown that the permeation properties are controlled not only by free volume fluctuations but also by energy barriers. A modified transport model taking into account the effect of the cohesive energy density on the energy barriers further improves the correlation significantly.