Impurity effects on the nucleation and growth of primary Al-3(Sc,Zr) phase have been investigated in high purity Al alloys and commercial purity Al alloys, respectively. In the case of high purity Al alloys, primary Al-3(Sc,Zr) phases were found to be pushed to grain boundaries ahead of the solidification front. Such type of primary Al-3(Sc,Zr) phase did not contribute to the heterogeneous nucleation, and thereby the grain refinement of Al alloys. In the case of commercial purity Al alloys, the presence of Fe, Si, Cu, Mg, Ti, and other impurities significantly enhanced the heterogeneous nucleation of primary Al-3(Sc,Zr) phase. Most primary Al-3(Sc,Zr) phases were found to be located within the alpha-Al matrix, and kept an identical orientation relationship with the alpha-Al matrix. Furthermore, the presence of the impurities also changed the growth mode on the primary Al-3(Sc,Zr) phase. In the case of commercial purity Al alloys, a peritectic to eutectic reaction was induced due to the presence of the impurities. A layered growth was observed leading to a narrow particle size distribution. In contrast, in the case of high purity Al alloys, a featureless structure was observed. This investigation demonstrates that impurities and their concentrations are important factors affecting the nucleation and growth of primary Al-3(Sc,Zr) phases, and thereby for the successful grain refinement in Al-based alloys.