The Hybrid Ground Source Heat Pump (GSHP) systems combine the renewable geothermal energy and cooling tower for rejecting the cooling load, which is often adopted for high cooling demand. Model based control can be limited due to variations in ambient conditions, ground-loop heat exchanger (GHE) and equipment characteristics, cost and reliability of sensors. A self-optimizing control scheme is proposed for efficient operation of the hybrid GSHP based on Extremum Seeking Control (ESC), with feedback of the total power consumption and the control inputs of the relative flow rate of cooling tower and the water pump speed. The cooling capacity of the heat pump regulates the evaporator leaving water at 7 degrees C. A Modelica based dynamic simulation model is developed for a Hybrid GSHP system, with the vertical GHE model adopted from Modelica Buildings Library. The transient heat transfer is implemented with a finite volume method inside and outside the borehole. The proposed ESC scheme is evaluated under the scenarios of fixed cooling load, ramp change in the evaporator inlet water temperature, diurnal sinusoidal cycle of air wet-bulb temperature, and realistic ambient and cooling load condition. Simulation results show the proposed ESC strategy effectively achieves nearly optimal efficiency without the need for plant model. (C) 2015 Elsevier Ltd. All rights reserved.