The mixing of granular particles in rotating wavy wall tumblers is simulated by the discrete element method. The rotating wavy wall is modeled by sinusoidal traveling waves on a fixed circle. Six wavenumbers (k(lambda) = 0, 3, 5, 7, 9, 11) under three rotating velocities (omega(d) = 0.1, 0.5, and 1.0 pi/s) are simulated for detailed analysis. The mixing characteristics are explored by using the difference of dimensionless particle concentrations and the fractal dimension of internal mixing interface, as well as mechanical energies, etc. It is found that the wavy wall tumbler performs more effectively for particle mixing enhancement than the circular tumbler, especially under low rotating velocities. The underlying mechanisms are explained; i.e., the rotating wavy wall can produce a periodic oscillating motion upon the conventional rotating motion of the tumbler. The oscillating motion is especially useful for enhancing mixing under low rotating velocities, although it seems to be overly effective under high rotating velocities since it makes the transition to the centrifugal regime earlier. The results improve the knowledge of particle mixing characteristics in wavy tumblers and are very useful for designing effective mixers.