The affinity of Nd and Dy to phosphate containing mineral phases was investigated in this study in an Al2O3-CaO-MgO-P2O5-SiO2 slag system. The slags originate from pyrometallurgical recycling of NdFeB magnets. Without addition of phosphate, the distribution of Nd and Dy is more or less arbitrary in the silicate matrix. Addition of phosphate leads to the formation of a silico-phosphate in an early stage of the crystallization process, which scavenges the majority of Nd and Dy. This is due to the strong affinity of these elements to phosphate structures. This phenomenon could be used to produce a rare earth element concentrate by classical mineral beneficiation.