Nitrogen-doped carbons can effectively stabilize noble metal particles to achieve high catalytic performances. However, the metallic impurities in carbon nanomaterials, e.g. the residual growth catalysts of carbon nanotubes (CNTs), have some unforeseen effects on the catalysis involving nanocarbons. Herein, we demonstrate that the residual growth catalysts of N-doped CNTs (NCNTs) may significantly deactivate Pd catalysts for the hydrogenation of nitrobenzene. Through high-resolution transmission electron microscopy and CO-stripping, it was determined that the N dopants improved the dispersion of Pd nanoparticles. However, the iron, at ppm level, in residual catalysts encapsulated inside NCNTs can transfer onto the surface of Pd to block active sites so that the activity of Pd/NCNTs was much lower than that of Pd/CNTs. The similar effect was observed for most of the common metallic impurities in carbon materials, including Co, Ni, Mn, Cr, Cu, Zn, Mo, Al and Mg. To exploit the N-doped carbons, we deposited N-doped carbon layers on purified CNTs through pyridine pyrolysis and then supported Pd nanoparticles. By this means, the activity of Pd for nitrobenzene hydrogenation was improved by 3.85 folds compared to conventional CNTs, emphasizing the importance of controlling impurities in N-doped carbon materials for high performance catalysts.