Shape memory thermoplastic polyurethane (SMTPU) containing isophorone diisocyanate (IPDI) in hard segment has excellent shape recoverability even after large strain deformation. However, the underlying mechanism of shape recovery remains unclear. In this study, 1,4-phenylene diisocyanate (PPDI) in the polyurethane is gradually substituted by IPDI, and multiscale effects are examined by normal and dichroic Fourier transform infrared spectroscopy (FTIR), small angle X-ray scattering (SAXS), single-molecule force spectroscopy (SMFS), and mechanical test. Contradictory to the traditional conclusion, the degree of microphase separation decreases as the content of IPDI increases, while the macroscopic shape recoverability is largely improved. With dichroic FTIR and SAXS, we find that the morphology of hard phases changed from lamellar-like to fibrillar-like, which is more stable under stretching. SMFS experiments discover that IPDI could increase the elasticity of polymer chain and could endow the hard phases with "elastic" under stress. With these two factors, we are able to explain the high recoverability of the SMTPU containing IPDI.