We present a method for computing an approximately optimal policy for the control of a hydroelectric reservoir with random inflows and concave, piecewise linear revenues from electricity sales. Our approach uses neurodynamic programming to approximate the future value function by a neural network. Our approximation architecture, based on the feedforward network, gives very smooth approximate functions, allowing the use of a coarse discretization of the state and inflow variables in the training step of the neural functions. Our model takes into account the head variations on the turbine efficiency and assumes the water flows at a steady rate during each period of the planning horizon, while related models in the literature have made less realistic assumptions of constant head or that the natural inflows were unusable until the next period. Moreover, we extend previous results on the concavity of the expected future rewards as a function of the potential energy in the reservoir and on the structure of optimal decision rules.