The effect of diffusion on formation of self-assembled CdSe quantum dots (SAQDs) was studied in this paper. Two kinds of diffusion were observed during the growth of CdSe layer. Surface diffusion was confirmed by atomic force microscopy (AFM). Interdiffusion was verified by the photoluminescence (PL) and X-ray diffraction measurement. The surface diffusion might help to form quantum dots. However, the interdiffusion could prevent the formation of quantum dots.