The influence of the backbone on the photoinduced birefringence was investigated in a poly(malonic ester) containing a p-cyanoazobenzene group as a side-chain. The photoinduced birefringence was successively measured in the fifteen cycles of the recording, relaxation, and erasing processes. The linearly polarized light, with a polarization direction of 45degrees relative to the analyzer, was used as the pumping beam during the first first five cycles, and the same pumping beam at negative 45degrees was used during the next 10 cycles. When the polarization direction of the pumping beam was changed perpendicularly, a very interesting birefringence curve was observed in the recording and relaxation processes, which could be well explained by our model considering the reorientation of the backbone caused by the elastic force between the backbone and side-chain. Consequently, we confirmed the fact that the relaxation behavior of the photoinduced birefringence is closely associated with the reorientation of the backbone.