Rylene dimides are widely used as the building blocks for n-type semiconducting polymers due to the tunable electronic properties. To elucidate their potentials as the electron acceptors in all-polymer solar cells, systematic comparisons of the properties among the derivatives are necessary. Herein, we used perylene diimide (PDI), naphthalene diimide (NDI), and naphthodithiophene diimide (NDTI) with the same alkyl chains combined with dithienothiophene (DTT) unit to obtain three polymer acceptors PPDI-DTT, PNDI-DTT, and PNDTI-DTT, respectively. Light absorption, carrier mobility, film morphology, and molecular orientation were characterized and compared. The photovoltaic devices based on PPDI-DTT, PNDI-DTT, and PNDTI-DTT achieved power conversion efficiency (PCE) of 3.49, 2.50, and 5.57%, respectively, in combination with BDDT as the donor polymer. The high performance of PNDTI-DTT was attributed to the strong absorption profile in the near-infrared (NIR) region, high and balanced electron and hole mobilities, and the preferable face-on orientation for the polymer chains in the blend films. The results indicate that NDTI is a promising building block to construct n-type photovoltaic polymers, and higher photovoltaic performance is anticipated with the further development of novel NDTI-based polymers.