Photoresponsive polymers convert a light stimulus input into a mechanical output (work). Photoinduced conformational changes, such as within azobenzene, dictate molecular-level distortions that summate into a macroscopic strain, which often manifests as a shape change or motion. The transduction of the molecular-level response to a macroscale effect is regulated by mesoscopic features, such as chain packing, free volume, and local molecular order-factors which depend on chemical composition as well as the process history of the material. Herein, we demonstrate the ability to widely tailor the photomechanical response of a photoresponsive polymer by manipulating the energy state of the glass, rather than formulating new chemical compositions. Physical aging increases the density of the glass, reduces local free volume, and thus reduces the minima in local conformation space, thereby strongly influencing the azobenzene photochemistry (trans-cis-trans isomerization). The subsequent change in the energy landscape of the system reduces the fraction of azobenzene able to undergo reconfiguration as well as increases the probability that those photoinduced conformations will relax back to the initial local environment. The result is a tuning of the magnitude of macroscopic strain and the ability to shift from shape fixing to shape recovery, respectively.