We study the Rayleigh instability of polystyrene (PS) thin films coated in the nanopores of anodic aluminum oxide (AAO) templates. After thermal annealing, the surface of the PS thin films undulates and the nanostructures transform from nanotubes to Rayleigh-instability-induced nanostructures (short nanorods with encapsulated air bubbles). With longer annealing times, the nanostructures further transform to nanorods with longer lengths. PS samples with two different molecular weights (24 and 100 kg/mol) are used, and their instability transformation processes are compared. The morphology diagrams of the nanostructures at different stages are also constructed to elucidate the mechanism of the morphology transformation.