Structures of CsTe2O6-x phases were investigated by single-crystal X-ray diffraction and neutron powder diffraction. Stoichiometric CsTe2O6 is a mixed-valence Cs2Te4+Te36+O12 compound with a rhombohedral pyrochlore-type structure where there is complete order of Te4+ and Te6+. On heating, this compound develops significant electrical conductivity. As CsTe2O6 becomes oxygen deficient above 600 degrees C, the rhombohedral pyrochlore-type structure is replaced by a cubic pyrochlore-type structure with disordered Te4+/Te6+ and oxygen vacancies. However, for CsTe2O6-x phases prepared at 500 degrees C, the observed pyrochlore-type structure has orthorhombic symmetry. The Te4+ and O vacancies are all on chains running along the b axis, and the maximum value of x observed is about 0.3. At still higher values of x a new compound was discovered with a structure related to that reported for Rb4Te34+Te56+O23.