Magic-sized clusters (MSCs) of semiconductor are typically defined as specific molecular-scale arrangements of atoms that exhibit enhanced stability. They often grow in discrete jumps, creating a series of crystallites, without the appearance of intermediate sizes. However, despite their long history, the mechanism behind their special stability and growth remains poorly understood. It is particularly difficult to explain experiments that have shown discrete evolution of MSCs to larger sizes well beyond the "cluster" regime and into the size range of colloidal quantum dots. Here, we study the growth of MSCs, including these larger magic-sized CdSe nanocrystals, to unravel the underlying growth mechanism. We first introduce a synthetic protocol that yields a series of nine magic-sized nanocrystals of increasing size. By investigating these crystallites, we obtain important clues about the mechanism. We then develop a microscopic model that uses classical nucleation theory to determine kinetic barriers and simulate the growth. We show that magic-sized nanocrystals are consistent with a series of zinc-blende crystallites that grow layer by layer under surface-reaction-limited conditions. They have a tetrahedral shape, which is preserved when a monolayer is added to any of its four identical facets, leading to a series of discrete nanocrystals with special stability. Our analysis also identifies strong similarities with the growth of semiconductor nanoplatelets, which we then exploit to further increase the size range of our magic-sized nanocrystals. Although we focus here on CdSe, these results reveal a fundamental growth mechanism that can provide a different approach to nearly monodisperse nanocrystals.