Young’s modulus of amorphous polymers around the glass-to-rubber transition zone can be separated into two component functions (R and G) through a modified stress-optical rule (Inoue et al. Macromolecules 1991, 24, 5670) with the simultaneous measurements of the strain-induced birefringence and the stress. In the present study, the two component functions are compared among more than ten polymers, and their relation to chemical structure is investigated. The Rouse segment size is estimated from the limiting modulus at high frequencies of the R component, which can be related to the chain orientation. The molecular weight of the Rouse segment agreed with that of the geometrically evaluated Kuhn segment. The G component that originates the glassy nature of amorphous polymers is grossly similar among studied polymers but depends on chemical structure if compared in detail. The distinction found in shapes of the G component could be related to the number of effective bonds per Rouse segment. This result suggests that the shape of segmental dispersion (G component) is related to the distribution of internal modes of the Rouse segment.