The high temperature thermal stability of the ultrafine-grained (UFG) microstructures in low stacking-fault-energy silver was studied by differential scanning calorimetry (DSC). The UFG microstructures in two samples having purity levels of 99.995 and 99.99 at.% were achieved by four passes of equal-channel angular pressing at room temperature. The defect structure was studied by electron microscopy, X-ray line profile analysis, and positron annihilation spectroscopy before and after the exothermic DSC peak related to recovery and recrystallization. The heat released in the DSC peak was correlated to the change of defect structure during annealing. It was found for both compositions that a considerable fraction of stored energy (similar to 15-20 %) was retained in the samples even after the DSC peak due to the remaining UFG regions and a large density of small dislocation loops in the recrystallized volumes. The larger impurity level in Ag yielded a higher temperature of recrystallization and a lower released heat. The latter observation is explained by the much lower vacancy concentration before the DSC peak which is attributed to the segregation of dopants at grain boundaries resulting in a smaller free volume in the interfaces.