Hydroprocessing technology is critical to reducing the sulfur and nitrogen content of complex heavy petroleum fractions, including vacuum gas oil (VGO) to increase the product value, to meet EPA regulations, and to avoid hampering the downstream processing. The catalyst design for hydroprocessing of VGO is of key importance, as the catalyst must perform multiple tasks in a complex and corrosive mixture. In this work, a combined approach is discussed that involves testing state-of-the-art catalysts for hydrodenitrogenation (HDN) and characterizing real VGO samples before and after HDN, using the advanced characterization method of ion mobility-mass spectrometry (IMMS). While other solvent systems have been reported, it is found that dichloromethane with 0.05% (v/v) trifluoroacetic acid is efficient for ionizing carbazole-type species, in addition to other nitrogen species. Comparing samples from three different catalysts reveal that catalysts A, B, and C can achieve similar performance for deep HDN. Interestingly, catalyst C is found to be more active than catalyst A and catalyst B for moderate HDN. This led to testing of a dual catalyst system using a layered bed of catalyst C and catalyst A that results in lower nitrogen concentrations than can be achieved using the individual catalysts. Overall, the combined approach expedited catalyst design and optimization for VGO hydroprocessing.