The dispersive mixing of particles suspended in Newtonian and viscoelastic fluids in a four-roll mill is studied by direct numerical simulations. A fictitious domain method is used to handle the particle motion. To quantify the mixing, a proper mixing distribution function is defined. The combined effect of fluid rheology and particle-particle/particle-wall hydrodynamic interactions is addressed. At variance with the Newtonian case where the particle distribution remains uniform, the viscoelasticity-induced migration leads to a significant segregation process. The effect of the Deborah number (the product of the fluid relaxation time and the roll angular velocity), shear-thinning, particle concentration, and size on the microstructure evolution is thoroughly investigated.