Fluorinated and chlorinated isoindigo-based polymers, F-IIDT and Cl-IIDT, were developed as the donor materials for photovoltaic cells. Although showing similar frontier orbital energy levels, both polymers exhibited distinct solar cell device performance, for example, power conversion efficiency of 4.60% for Cl-IIDT and 1.19% for F-IIDT. The investigation of photophysical properties demonstrated that the strong preaggregation of F-IIDT was formed in solution due to increased noncovalent interactions between the backbones of the polymer. Therefore, large crystal domains in blended thin films with no preferred crystallographic orientation were obtained. On the other hand, the introduction of chlorine atoms increased the torsional angle of the polymer backbone, resulting in reduced crystallization tendency. Films with less crystallinity and face-on polymer orientation contributed to better device performance of Cl-IIDT. Our research proved that the effects of halogenation are far more than the tuning of frontier energy levels. Phase separation in blended films, including size of the crystallized domains and molecular packing orientation, is also greatly influenced by halogenation.