In this paper, asymptotic swarm stability of fractional-order swarm systems in the presence of uniform time-delays is studied. By using a frequency domain approach, the necessary and sufficient condition on the values of time-delay is derived to guarantee the asymptotic swarm stability under a directed/undirected interaction graph. Numerical examples are given to confirm the obtained analytical results.