In this paper, a novel problem formulation for optimal coordination between distance relays and Directional Over-Current Relays (DOCRs) in series compensated systems is proposed. The integration of the series capacitor (SC) into the transmission line makes the coordination problem more complex. The main contribution of this paper is a new systematic method for computing the optimal second zone timing of distance relays and optimal settings of DOCRs, in series compensated and uncompensated transmission systems, which have a combined protection scheme with DOCRs and distance relays. In order to solve this coordination problem, which is a nonlinear and non-convex problem, a Modified Adaptive Particle Swarm Optimization (MAPSO) is employed. The new proposed method is supported by obtained results from a typical test case and a real power system network. (C) 2011 Elsevier Ltd. All rights reserved.