This paper addresses distributed optimal tracking control of multi-agent linear systems subject to external disturbances. The concept of differential game theory is utilized to formulate this distributed control problem into a multi-player zero-sum differential graphical game, which provides a new perspective on distributed tracking of multiple agents influenced by disturbances. In the presented differential graphical game, the dynamics and performance indices for each node depend on local neighbor information and disturbances. It is shown that the solution to the multi-agent differential graphical games in the presence of disturbances requires the solution to coupled Hamilton-Jacobi-Isaacs (HJI) equations. Multi-agent learning policy iteration (PI) algorithm is provided to find the solution to these coupled HJI equations and its convergence is proven. It is also shown that L-2-bounded synchronization errors can be guaranteed using this technique. An online PI algorithm is given to solve the zero-sum game in real time. A simulation example is provided to show the effectiveness of the online approach. (C) 2016 Elsevier Ltd. All rights reserved.