This paper considers the discrete-time version of Altafini's model for opinion dynamics in which the interaction among a group of agents is described by a time-varying signed digraph. Prompted by an idea from [3], exponential convergence of the system is studied using a graphical approach. Necessary and sufficient conditions for exponential convergence with respect to each possible type of limit states are provided. Specifically, under the assumption of repeatedly jointly strong connectivity, it is shown that 1) a certain type of two-clustering will be reached exponentially fast for almost all initial conditions if, and only if, the sequence of signed digraphs is repeatedly jointly structurally balanced corresponding to that type of two-clustering; 2) the system will converge to zero exponentially fast for all initial conditions if, and only if, the sequence of signed digraphs is repeatedly jointly structurally unbalanced. An upper bound on the convergence rate is provided. The results are also extended to the continuous-time Altafini model.