The unique properties of magnetic nanoparticles (MNPs), coupled with versatile surface engineering techniques, have led to a rising class of screening methods that enable separation of specific cell populations from complex biological samples. The growing sophistication and efficiency of these methods have far reaching implications for both fundamental research and clinical applications. In this study, the synthesis and surface engineering of MNPs is reviewed. Here, a model is introduced to illustrate how MNP morphology and particle-particle interactions influence magnetization, which is a key consideration in designing and selecting MNPs for efficient cell separations. Building upon these themes, immunomagnetic assays for capturing, isolating, and characterizing rare cell types from complex biological mixtures are reviewed. Although the focus of this study is on circulating tumor cells, these same techniques can be applied in screening for other rare cells of interest, such as various stem cell populations. In conclusion, current challenges and future directions for magnetic-nanomaterial-based cell screening systems are discussed.