Rigid molecule atomistic lattice dynamics calculations have been performed to predict the phonon spectra of the four polymorphs of carbamazepine, and these calculations predict that there should be differences in the spectra of all four forms. Terahertz spectra have been measured for forms I and III, and there are clearly different features between polymorphs' spectra, that are accentuated at low temperature. While carbamazepine adopts the same hydrogen bonded dimers in all of its known polymorphs, the calculations show that differences in packing arrangements of the dimers lead to changes in the frequency ranges for each type of hydrogen bond vibration, giving a physical explanation to the observed differences between the spectra. Although the agreement between calculated and observed spectra does not allow a definitive characterization of the spectra, it is possible to make tentative assignments of many of the observed features in the terahertz region for the simpler form III; we can only make some tentative assignments of specific modes in the more complex spectrum of form I. While harmonic rigid molecule lattice dynamics shows promise for understanding the differences in spectra between polymorphs of organic molecules, discrepancies between observed and calculated spectra suggest areas of improvement in the computational methods for more accurate modeling of the dynamics in molecular organic crystals.