We report a three-orders-of-magnitude variation of carrier lifetimes in exotic crystalline phases of gold nanoclusters (NCs) in addition to the well-known face-centered cubic structure, including hexagonal close-packed (hcp) Au-30 and body-centered cubic (bcc) Au-38 NCs protected by the same type of capping ligand. The bcc Au-38 NC had an exceptionally long carrier lifetime (4.7 microseconds) comparable to that of bulk silicon, whereas the hcp Au-30 NC had a very short lifetime (1 nanosecond). Although the presence of ligands may, in general, affect carrier lifetimes, experimental and theoretical results showed that the drastically different recombination lifetimes originate in the different overlaps of wave functions between the tetrahedral Au-4 building blocks in the hierarchical structures of these NCs.