The vacancy-enhanced contact friction of graphene is mainly attributed to the vacancy-enhanced out-of-plane deformation flexibility of the graphene and the climbing of the tip out of the vacancy trap (which actually acts as a step edge). However, this mechanism does not apply for explaining the enhanced friction caused by small-sized vacancies that are unable to accommodate the tip, such as single vacancy and double vacancies, which also commonly exist in the graphene. In the present study, by performing a set of classic molecular dynamics simulations, we demonstrated that the double-vacancy defect in graphene substantially enhanced the contact friction when the tip slides over it and the pinning effect of the reconstructed lattice of the double-vacancy defect with atoms at the bottom of the tip dominated such an influence. The underlying mechanism of such an atomic pinning effect and the influence of the normal load, sliding direction, and the sliding velocity were unveiled by analyzing the obtained friction evolution and the atomic configuration and interaction between the tip and the graphene. We believe that the findings presented in this study complete the state-of-art understanding of the nanoscale friction behaviors of vacancy-defected graphene, which is essential for the implementation of their potential control.