Using a counter rotating parallel plate shear flow cell, the shape relaxation of deformed droplets in a quiescent matrix is studied microscopically. Both the effects of geometrical confinement and component viscoelasticity are systematically explored at viscosity ratios of 0.45 and 1.5. The flow conditions are varied from a rather low to a nearly critical Ca number. Under all conditions investigated, viscoelasticity of the droplet phase has no influence on shape relaxation, whereas matrix viscoelasticity and geometrical confinement result in a slower droplet retraction. Up to high confinement ratios, the relaxation curves for ellipsoidal droplets can be superposed onto a master curve. Confined droplets with a sigmoidal shape relax in two stages: the first consists of a shape change to an ellipsoid with a limited amount of retraction, and the second is the retraction of this ellipsoid. The latter stage can be described by means of one single relaxation time that can be obtained from the relaxation of initially ellipsoidal droplets. The experimental results are compared to the predictions of a recently published phenomenological model for droplet dynamics in confined systems with viscoelastic components (Minale et al., Langmuir 26:126-132, 2010). However, whereas the model predicts additive effects of geometrical confinement and component viscoelasticity, the experimental data reveal more complex interactions.