We investigate the numerical solution of boundary control problems with elliptic partial differential equations by the hp-finite element method. We prove exponential convergence with respect to the number of unknowns for an a priori chosen discretization. Here, we have to prove that derivatives of arbitrary order of the solution belong to suitably chosen weighted Sobolev spaces. This result relies on the assumption that the number of switching points of the optimal control is finite. Numerical experiments confirm the theoretical findings.