To reduce the aggregation of graphene, Ag particles were introduced in the graphene nanosheets by chemical reduction method, and the silver matrix composite reinforced with Ag-doped graphene nanosheets were fabricated by powder metallurgy. The synthesized Ag-doped graphene were characterized by scanning electron microscopy (SEM), transmission electron microscopy (TEM), Raman spectroscopy (Raman) and X-ray photoelectron spectroscopy (XPS), and the relative density, hardness along with electrical conductivity of the silver matrix composite reinforced with Ag-doped graphene were measured as well. The results show that silver ions and graphene oxide can be effectively reduced by ascorbic acid simultaneously, most of oxygen functional groups on the Ag-doped graphene sheets can be removed dramatically, and Ag-doped graphene inhibit the graphene aggregation effectively. Compared with pure Ag, the hardness of the 0.5 wt% Ag-doped graphene reinforced sliver matrix composite is increased by 35.1% and the electrical conductivity reaches up to 98.62% IACS (International Annealed Copper Standard). However, excessive addition of Ag-doped graphene is not beneficial for the improvement on the hardness and electrical conductivity of silver matrix composite.