Consider a general nonlinear optimal control problem in finite dimension, with constant state and/or control delays. By the Pontryagin maximum principle, any optimal trajectory is the projection of a Pontryagin extremal. We establish that, under appropriate assumptions which are essentially sharp, Pontryagin extremals depend continuously on the parameters delays, for adequate topologies. The proof of the continuity of the trajectory and of the control is quite easy; however, for the adjoint vector, the proof requires a much finer analysis. The continuity property of the adjoint vector with respect to the parameter delays opens a new perspective for the numerical implementation of indirect methods, such as the shooting method.