Search for a new geometrically frustrated lattice is a great challenge. Herein, we report on a successful synthesis of two new layered compounds BiOCu2(XO3)(SO4)(OH)center dot H2O [X = Te (1) and Se (2)] with a new type of geometrically frustrated lattice (i.e., the octa-kagome lattice) between kagome and star motifs. Magnetic measurements confirmed that 1 exhibits a spin gap ground state, while 2 possesses a typical antiferromagnetic ordering at low-temperature. Such different magnetic behaviors between two isostructural compounds are suggested to originate from a slightly structural modification induced by nonmagnetic XO3 anionic groups. Theoretical simulations suggest that the origin of gapped ground state in 1 may be due to the dimerization of Cu2+ ions, while 2 may break the limiting of such dimerization, leading to an antiferromagnetic ordering.