The optical dephasing, caused by vibrational modulation of interactions of optical centers with defects is studied. This mechanism is essential for crystals with a substantial amount of defects, being more efficient at low temperatures than the Raman mechanism, caused by scattering of acoustic phonons at optical centers. A theory is developed, which allows one to find averaged widths and shapes of spectral holes in these systems dependending on the temperature T and concentration c of defects. The shape of the hole is found to be close to a Lorentzian, and at intermediate temperatures its width is proportional to T-3, and strongly depends on c (for charged point defects as c(8/3)).