A detailed computational study of the potential of mean force between a pair of spherical particles dissolved in a homopolymer melt has been performed using microscopic liquid state theory. The role of particle-to-monomer diameter ratio, degree of polymerization, strength and spatial range of monomer-particle attractions, and direct interfiller attractions has been established. Beyond the small particle regime, the potential of mean force scales linearly with the particle-to-monomer diameter ratio. This simple scaling allows the construction of master curves and the quantification of material specific aspects independent of the filler-to-monomer diameter ratio. For hard-sphere fillers, four general categories of polymer-mediated organization are found: contact aggregation due to depletion attraction, segment level tight particle bridging, steric stabilization due to thermodynamically stable "bound polymer layers", and "tele-bridging" where distinct adsorbed layers coexist with longer range bridging. The conditions on the strength and spatial range of monomer-particle attractive interactions that define these different modes of organization have been established. Direct interparticle van der Waals attractions favor contact aggregation and thus compete with the rich polymer-mediated behavior. As the direct attractions increase in strength, the globally stable noncontact bridging configuration is gradually destabilized and replaced by contact aggregation as the most favored state of packing. However, bridging states often remain as metastable local minima. Steric stabilization systems are much less affected by direct interfiller attractions due to the thermodynamic stability of distinct bound polymer layers. This suggests design rules for achieving good particle dispersion. In addition, the interesting possibility is raised that sterically stabilized nanofillers may crystallize in a homopolymer matrix at relatively low volume fractions. Our results have implications for nonequilibrium phenomena such as gelation or filler network formation and kinetic stabilization via large repulsive barriers, which are qualitatively discussed.