Azobenzene is one of the most famous photoresponsive chromophores, and it repeatedly photo-isomerizes. In this study, main-chain type polyazobenzenes bearing chiral substituents in the side-chain are synthesized by the Suzuki-Miyaura coupling method. The chiral aggregation process of main-chain type polyazobenzene is investigated. Then, intermolecular interaction of these polymers allows the formation of chiral aggregates as higher order structures. Trans-to-cis photoisomerization of the polymers was confirmed with in situ UV-vis absorption spectra. Development of pi-conjugation in the backbone can be an important factor in the design of main-chain type azobenzene polymers. In addition, we induced light-driven changes in the chiroptical activity. The microstructure of the polymers is examined with synchrotron radiation X-ray diffraction and a technique to define "chiral photochromism".