Structural and vibrational properties of orthorhombic BaTe2O6, a mixed valence tellurium compound, have been investigated by in situ synchrotron X-ray diffraction (XRD) studies up to 16 GPa and Raman spectroscopy up to 37 GPa using a diamond-anvil cell. The structure of orthorhombic BaTe2O6 has layers of [Te2O6](2-), formed by TeO6 octahedra and TeO5 square pyramids and Ba' ions stacked alternately along the (010) direction. A reversible pressure-induced structural transformation from the ambient orthorhombic (Cmcm) to a monoclinic (P2(1)/m) structure is observed in both XRD and Raman spectroscopic investigations around 10 GPa. Ab initio calculations using density functional theory (DFT) corroborate this phase transition as well as the transition pressure. Both XRD and DFT calculations reveal that the highpressure monoclinic structure is closely related to the ambient pressure orthorhombic structure, and the transformation is accompanied by a slight rearrangement of the structural units. Pressure evolution of unit cell parameters of the ambient pressure phase reveals an anisotropic compressibility with maximum compression along the b-axis as compared to other crystallographic directions. The bulk modulus and its derivative are found to be B-0 = 88(2) GPa and B-0' = 4.1(7) from high-pressure experiments, while those calculated by DFT are B-0 = 123 GPa and B-0' = 4.74. Pressure evolution of Raman spectra indicates significant changes across the orthorhombic to monoclinic phase transformation at 9.1 GPa and no evidence of further structural changes up to 37 GPa. Raman mode frequencies, pressure coefficients, and Graneisen parameters in the low- and high-pressure phases of BaTe2O6 have been obtained for both monoclinic and orthorhombic phases. Finally, the mechanism for the instability of the orthorhombic phase under pressure is proposed.