In this paper, we report the divalent ions Mg2+, Zn2+, and Cu2+ as dopants to manipulate the crystal field asymmetry in a new cubic Y3.2Yb0.4Er0.08Al0.32F12 phase which features two different coordination sites for rare earth ions. X-ray powder diffraction, transmission electron microscopy, X-ray photoelectron spectroscopy, and fluorescence spectrophotometer were employed for the crystal structure analysis and luminescence performance investigation. Results reveal that the phase transition from cubic to orthogonal of Y3.2Yb0.4Er0.08Al0.32F12 is promoted by Mg2+ doping. Among Mg2+, Zn2+, and Cu2+, Mg2+ is found to be the most effective dopant for the upconversion performance enhancement. The crystal lattice structure asymmetry increases with the rising of Mg2+ doping concentration and reaches the peak when 6 mol% Mg2+ is introduced; correspondingly, the brightness of 408 and 654 nm emissions are strikingly enhanced over 20 times, and the lifetime of 540 nm emission is shortened to half of that in the Mg2+-free sample. The investigation results establish the understanding of divalent ions as dopants for adjusting upconversion luminescence performance and may be helpful for researchers to develop quick responsive upconversion luminescence materials.