FTIR spectroscopy has been used to probe the mobility of matrix-isolated species by monitoring the formation of dimers and higher-order multimers in a 1:1600 HBr doped xenon matrix as a function of both the initial deposition temperature and subsequent annealing treatment. Upon subsequent annealing, matrices initially deposited at lower temperatures produce larger concentrations of multimeric HBr species compared to matrices formed by initial deposition at higher temperatures. These results are interpreted in the context of the effect of the microscopic morphology of a matrix on the mobility of matrix-isolated species. Time-resolved studies of the formation of multimers allow an approximate diffusion coefficient to be determined for HBr at 50 K. These results are consistent with HBr mobility in these rare gas matrices being defect mediated, which is enhanced in more amorphous matrices formed at lower deposition temperatures.