The orthorhombic phase of GeSe, a structural analogue of layered SnSe (space group: Pnma), has recently attracted attention after a theoretical prediction of high thermoelectric figure of merit, zT > 2. The experimental realization of such high performance in orthorhombic GeSe, however, is still elusive (zT approximate to 0.2). The rhombohedral phase of GeSe, a structural analogue of GeTe (space group: R3m), previously stabilized at high pressure (2 GPa) and high temperature (1600 K), is promising due to its theoretically predicted ferroelectric instability and the higher earth abundance of Se compared to Te. Here, we demonstrate high thermoelectric performance in the rhombohedral crystals of GeSe, which is stabilized at ambient conditions by alloying with 10 mol % AgBiSe2. We show ultralow lattice thermal conductivity (kappa(L)) of 0.74-0.47 W/mK in the 300-723 K range and high zT approximate to 1.25 at 723 K in the p-type rhombohedral (GeSe)(0.9)(AgBiSe2)(0.1) crystals grown using Bridgman method. First-principles density functional theoretical analysis reveals its vicinity to a ferroelectric instability which generates large anomalous Born effective charges and strong coupling of low energy polar optical phonons with acoustic phonons. The presence of soft optical phonons and incipient ferroelectric instability in (GeSe)(0.9)(AgBiSe2)(0.1) are directly evident in the low temperature heat capacity (C-p) and switching spectroscopy piezoresponse force microscopy (SS-PFM) experiments, respectively. Effective scattering of heat carrying acoustic phonons by ferroelectric instability induced soft transverse optical phonons significantly reduces the kappa(L) and enhances the thermoelectric performance in rhombohedral (GeSe)(0.9)(AgBiSe2)(0.1) crystals.