We use single-particle tracking to investigate colloidal dynamics in hybrid assemblies comprising colloids enmeshed in a crosslinked polymer network. These assemblies are prepared using ice templating and are macroporous monolithic structures. We investigate microstructure-property relations in assemblies that appear chemically identical but show qualitatively different mechanical response. Specifically, we contrast elastic assemblies that can recover from large compressive deformations with plastic assemblies that fail on being compressed. Particle tracking provides insights into the microstructural differences that underlie the different mechanical response of elastic and plastic assemblies. Since colloidal motions in these assemblies are sluggish, particle tracking is especially sensitive to imaging artifacts such as stage drift. We demonstrate that the use of wavelet transforms applied to trajectories of probe particles from fluorescence microscopy eliminates stage drift, allowing a spatial resolution of about 2 nm. In elastic and plastic scaffolds, probe particles are surrounded by other particles-thus, their motion is caged. We present mean square displacement and van Hove distributions for particle motions and demonstrate that plastic assemblies are characterized by significantly larger spatial heterogeneity when compared with the elastic sponges. In elastic assemblies, particle diffusivities are peaked around a mean value, whereas in plastic assemblies, there is a wide distribution of diffusivities with no clear peak. Both elastic and plastic assemblies show a frequency independent solid modulus from particle tracking microrheology. Here too, there is a much wider distribution of modulus values for plastic scaffolds as compared to elastic, in contrast to bulk rheological measurements where both assemblies exhibit a similar response. We interpret our results in terms of the spatial distribution of crosslinks in the polymer mesh in the colloidal assemblies.