The integration of photochromic molecules into semiconducting polymer matrices via blending has recently attracted a great deal of attention, as it provides the means to reversibly modulate the output signal of electronic devices by using light as a remote control. However, the structural and electronic interactions between photochromic molecules and semiconducting polymers are far from being fully understood. Here we perform a comparative investigation by combining two photochromic diarylethene moieties possessing similar energy levels yet different propensity to aggregate with five prototypical polymer semiconductors exhibiting different energy levels and structural order, ranging from amorphous to semicrystalline. Our in-depth photo- chemical, structural, morphological, and electrical characterization reveals that the photoresponsive behavior of thin-film transistors including polymer/diarylethenes blends as the active layer is governed by a complex interplay between the relative position of the energy levels and the polymer matrix microstructure. By matching the energy levels and optimizing the molecular packing, high-performance optically switchable organic thin-film transistors were fabricated. These findings represent a major step forward in the fabrication of light-responsive organic devices.