Nonlinear optimal control and nonlinear H-infinity control are two of the most significant paradigms in nonlinear systems theory. Unfortunately, these problems require the solution of Hamilton-Jacobi equations, which are extremely difficult to solve in practice. To make matters worse, approximation techniques for these equations are inherently prone to the so-called 'curse of dimensionality'. While there have been many attempts to approximate these equations, solutions resulting in closed-loop control with well-defined stability and robustness have remained elusive. This paper describes a recent breakthrough in approximating the Hamilton-Jacobi-Bellman and Hamilton-Jacobi-Isaacs equations. Successive approximation and Galerkin approximation methods are combined to derive a novel algorithm that produces stabilizing, closed-loop control laws with well-defined stability regions. In addition, we show how the structure of the algorithm can be exploited to reduce the amount of computation from exponential to polynomial growth in the dimension of the state space. The algorithms are illustrated with several examples.