Dispersing solid fillers into a polymer matrix is a common strategy to enhance and tailor its properties. The polymer nanocomposites so obtained with fractal-like aggregates have exceptional rheological behavior that have long been exploited in the tire industry. However, because of the disparity of time and length scales, our understanding of the relation between nanocomposites structure and rheology remains far from complete. We propose in this work a mesoscopic model to address the dynamics and the rheology of aggregate nanocomposites. While aggregates are represented explicitly as groups of interacting particles, we use for the polymeric matrix an implicit description based on generalized Langevin and Stokes equations that captures the average effect of a viscoelastic medium. These two-level modeling allows us to simulate large systems containing dozens of aggregates. Focusing here on the linear viscoelastic properties of PNCs (polymer nanocomposites), we characterize the influence of aggregate size, volume fraction, rigidity, and polydispersity. We demonstrate that rigid aggregates systems display salient features of the phenomenology of PNCs, namely, slow relaxation in the stress relaxation response, non-Maxwell elastic response at low frequency, and supralinear dependence of their storage modulus with volume fraction. We also point out the critical role played by aggregate rigidity and show that for either flexible aggregates or well-dispersed nanofillers the effects are far less spectacular. Our findings should help in designing nanocomposites with enhanced mechanical properties.