In this paper an optimal control problem is considered, where the control variable lies in a measure space and the state variable fulfills an elliptic equation. This formulation leads to a sparse structure of the optimal control. In this setting we prove a new regularity result for the optimal state and the optimal control. Moreover, a finite element discretization based on [E. Casas, C. Clason, and K. Kunisch, SIAM J. Control Optim., 50 (2012), pp. 1735-1752] is discussed and a priori error estimates are derived, which significantly improve the estimates from that paper. Numerical examples for problems in two and three space dimensions illustrate our results.