SIAM Journal on Control and Optimization, Vol.40, No.4, 1011-1041, 2001
Rate of convergence for constrained stochastic approximation algorithms
There is a large literature on the rate of convergence problem for general unconstrained stochastic approximations. Typically, one centers the iterate theta(n) about the limit point (θ) over bar and then normalizes by dividing by the square root of the stepsize epsilon(n). Then some type of convergence in distribution or weak convergence of U-n, the centered and normalized iterate, is proved. For example, one proves that the interpolated process formed by the U-n converges weakly to a stationary Gaussian diffusion, and the variance of the stationary measure is taken to be a measure of the rate of convergence. See the references in [ A. Benveniste, M. Metivier, and P Priouret, Adaptive Algorithms and Stochastic Approximation, Springer-Verlag, Berlin, New York, 1990; L. Gerencer, SIAM J. Control Optim., 30 (1992), pp. 1200 1227; H. J. Kushner and D. S. Clark, Stochastic Approximation for Constrained and Unconstrained Systems, Springer-Verlag, Berlin, New York, 1978; H. J. Kushner and G. Yin, Stochastic Approximation Algorithms and Applications, Springer-Verlag, Berlin, New York, 1997; M. T. Wasan, Stochastic Approximation, Cambridge University Press, Cambridge, UK, 1969] for algorithms where the stepsize either goes to zero or is small and constant. Large deviations provide an alternative approach to the rate of convergence problem [ P Dupuis and H. J. Kushner, SIAM J. Control Optim., 23 ( 1985), pp. 675 696; P Dupuis and H. J. Kushner, SIAM J. Control Optim., 27 ( 1989), pp. 1108 1135; P Dupuis and H. J. Kushner, Probab. Theory Related Fields, 75 ( 1987), pp. 223 244; A. P Korostelev, Stochastic Recurrent Processes, Nauka, Moscow, 1984; H. J. Kushner and G. Yin, Stochastic Approximation Algorithms and Applications, Springer-Verlag, Berlin, New York, 1997]. When the iterates of the algorithm are constrained to lie in some bounded set, the limit point is frequently on the boundary With the exception of the large deviations type [ P Dupuis and H. J. Kushner, SIAM J. Control Optim., 23 ( 1985), pp. 675 696; P Dupuis and H. J. Kushner, Probab. Theory Related Fields, 75 ( 1987), pp. 223 244], the rate of convergence literature is essentially confined to the case where the limit point is not on a constraint boundary When the limit point is on the boundary of the constraint set the usual steps are hard to carry out. In particular, the stability methods which are used to prove tightness of the normalized iterates cannot be carried over in general, and there is the problem of proving tightness of the normalized process and characterizing the limit process. This paper develops the necessary techniques and shows that the stationary Gaussian diffusion is replaced by an appropriate stationary reflected linear diffusion, whose variance plays the same role as a measure of the rate of convergence. An application to constrained function minimization under inequality constraints q(i) (x) less than or equal to0, i less than or equal to p, is given, where both the objective function and the constraints are observed in the presence of noise. The iteration is on both the basic state variable and a Lagrange multiplier, which is constrained to be nonnegative. If a limit multiplier value for an active constraint is zero, then the classical method for computing the rate cannot be used, but ( under appropriate conditions) it is a special case of our results. Rate of convergence results are important because, among other reasons, they immediately yield the advantages of iterate averaging methods, as noted in [ H. J. Kushner and G. Yin, Stochastic Approximation Algorithms and Applications, Springer-Verlag, Berlin, New York, 1997].
Keywords:stochastic approximation;constrained stochastic approximation;rate of convergence;recursive algorithms;weak convergence