We report the synthesis and characterizations of a new FeSe-based compound CsFe4-delta Se4, which is closely related to alkali intercalated FeSe superconductors while exhibits distinct features. It does not undergo phase separation and antiferromagnetic transition. Powder neutron diffractions, electron microscopy and high-angle annular-dark-field images confirm that CsFe4-delta Se4 possesses an ordered Cs arrangement as root 2 x root 2 superstructure, evidencing a B-centered orthorhombic lattice with a space group of Bmmm. The temperature-dependent powder neutron diffractions indicate no structural and magnetic transition from 320 to 5 K. In contrast to the symmetry-breaking in FeSe, this phase naturally possesses the orthorhombic symmetry even at room temperature. DFT calculations and transport measurements reveal a novel Fermi surface geometry with two electron-like sheets centered on Gamma point and intermediate density of states at the Fermi level comparing with the value of FeSe and the superconducting A(x)Fe(2)Se(2).