We prepared ferrites doped with different rare earth elements (La, Ce, Sm, Gd, Dy) by ball-milling the oxides of these metals and using double sintering technique. The so-prepared ferrites have been first characterized by Xray diffraction. We investigated the effect of rare earth doping on the spinel structure. The substituted rare earth ions, which are larger than the pristine ions, cause a different amount of distortion in the octahedral and tetrahedral sites of the ferrite lattice, which is correlated with their different size. This distortion in the spinel lattice induces different polarization effects when the materials are hit by incident electromagnetic radiation. The vibrational response of the prepared materials has been investigated by means of Raman and FTIR spectroscopy. The collected spectra show the fundamental modes as predicted by factor group theory. The dependency of Raman shift on rare earth doping affects the optical and magnetic properties of the material.