Although high power conversion efficiencies (PCE) have already been demonstrated in conventional structure polymer solar cells (PSCs), the development of high performance inverted structure polymer solar cells is still lagging behind despite their demonstrated superior stability and feasibility for roll-to-roll processing. To address this challenge, a detailed study of solution-processed, inverted-structure PSCs based on the blends of a low bandgap polymer, poly(indacenodithiophene-co-phananthrene-quinoxaline) (PIDT-PhanQ) and [6,6]-phenyl-C71-butyric acid methyl ester (PC71BM) as the bulk heterojunction (BHJ) layer is carried out. Comprehensive characterization and optical modeling of the resulting devices is performed to understand the effect of device geometry on photovoltaic performance. Excellent device performance can be achieved by optimizing the optical field distribution and spatial profiles of excitons generation within the active layer in different device configurations. In the inverted structure, because the peak of the excitons generation is located farther away from the electron-collecting electrode, a higher blending ratio of fullerene is required to provide higher electron mobility in the BHJ for achieving good device performance.