The effect of substituting iron and zinc for cobalt in CaBaCo4O7 was investigated using neutron diffraction and X-ray absorption spectroscopy techniques. The orthorhombic distortion present in the parent compound CaBaCo4O7 decreases with increasing the content of either Fe or Zn. The samples CaBaCo3ZnO7 and CaBaCo4-xFexO7 with x >= 1.5 are metrically hexagonal, but much better refinements in the neutron diffraction patterns are obtained using an orthorhombic unit cell. The two types of substitution have opposite effects on the structural and magnetic properties. Fe atoms preferentially occupy the sites at the triangular layer. Thus, the replacement of Co by Fe suppresses the ferrimagnetic ordering of the parent compound, and CaBaCo4-xFexO7 (0.5 <= x <= 2) samples are antiferromagnetically ordered following a new propagation vector k = (1/3,0,0). However, the Zn atoms prefer occupying the Kagome layer, which is very detrimental for the long-range magnetic interactions giving rise to a magnetic glass-like behavior in the CaBaCo3ZnO7 sample. The oxidation states of iron and zinc are found to be 3+ and 2+, respectively, independently of the content, as confirmed by X-ray absorption spectroscopy. Therefore, the average Co oxidation state changes accordingly with the Fe3+ or Zn2+ doping. Also, X-ray absorption spectroscopy data confirm the different preferential occupation for both Fe and Zn cations. The combined information obtained by neutron diffraction and X-ray absorption spectroscopy indicates that cobalt atoms can be either in a fluctuating Co2+/Co3+ valence state or, alternatively, Co2+ and Co3+ ions being randomly distributed in the lattice. These results explain the occurrence of local disorder in the CoO4 tetrahedra obtained by EXAFS. An anomaly in the lattice parameters and an increase in the local disorder are observed only at the ferrimagnetic transition for CaBaCo4O7, revealing the occurrence of local magneto-elastic coupling.