To optimize azobenzene-containing materials for applications such as rewritable waveguides and holographic data storage, it is imperative to understand the effect of the azobenzene structure on the photoresponse of the material. Supramolecular materials, in which a complexed photoactive azobenzene controls the motion or other properties of a passive polymer, are uniquely convenient for studying the impact of specific chemical modifications. Here, we use polarization modulation infrared structural absorbance spectroscopy (PM-IRSAS) to study hydrogen-bonded supra molecular azobenzene complexes using poly(4-vinylpyridine) (P4VP) as a model polymer. We show that changing the tail group from hydrogen (A(H)) to cyano (A(cN)) induces greater angular redistribution of the chromophores and, remarkably, provokes P4VP pyridine ring orientation. Increasing the degree of complexation decreases the saturated orientation of both A(H) and AcN, whereas for P4VP/AcN it increases the pyridine orientation as well as the orientation stability of both components. These results explain the contrasting photoinduced birefringence behavior previously observed for these complexes and identify azo-azo intermolecular interactions as the main reason. To our knowledge, this is the first molecular-level spectroscopic analysis of the contrasting contributions of azobenzenes to the photoorientation of supramolecular azopolymer complexes and the first report of the large impact of small molecular changes on the capacity of azo dyes to transfer light-induced orientation to a photopassive polymer.