To synthesize the optimal control strategies of nonlinear systems on infinite horizon while subject to mixed equality and inequality constraints has been a challenge to control engineers. This paper regards it as a problem of finite-time optimization in infinite-horizon control then devises a reinforcement learning agent, termed as the Adaptive Optimal Control (AOC) agent, to carry out the finite-time optimization procedures. Adaptive optimal control is in the sense of activating the finite-time optimization procedure whenever needed to improve the control strategy or adapt to a real-world environment. The Nonlinear Quadratic Regulator (NQR) is shown a typical example that the AOC agent can find out. The optimality conditions and adaptation rules for the AOC agent are deduced from Pontryagin's minimum principle. The requirements for convergence and stability of the AOC system are shown. (C) 2012 Elsevier Ltd. All rights reserved.