Simulation-Based Learning of Cost-To-Go for Control of Nonlinear Processes

Jong Min Lee; Jay H. Lee

Korean Journal of Chemical Engineering, Vol.21, No.2, 338-344, March, 2004

DOI10.1007/BF02705417 Export Citation

Simulation-Based Learning of Cost-To-Go for Control of Nonlinear Processes

Jong Min Lee , and Jay H. Lee^†

311 Ferst Dr. NW, School of Chemical and Biomolecular Engineering,, Georgia Institute of Technology, Atlanta, GA 30332-0100, USA

E-mail:

In this paper, we present a simulation-based dynamic programming method that learns the ‘cost-to-go’ function in an iterative manner. The method is intended to combat two important drawbacks of the conventional Model Predictive Control (MPC) formulation, which are the potentially exorbitant online computational requirement and the inability to consider the future interplay between uncertainty and estimation in the optimal control calculation. We use a nonlinear Van de Vusse reactor to investigate the efficacy of the proposed approach and identify further research issues.

Keywords:Nonlinear Model Predictive Control;Dynamic Programming;Stochastic Optimal Control;Reinforcement Learning;Neuro-dynamic Programming;Function Approximation

[References]

Astrom KJ, Helmersson A, Comp. Maths. Appl., 12A, 653 (1986)
Astrom KJ, Wittenmark B, "Adaptive Control," Addison-Wesley (1989)
Bellman RE, "Dynamic Programming," Princeton University Press, New Jersey (1957)
Bemporad A, Morari M, Automatica, 35(3), 407 (1999)
Bertsekas DP, "Dynamic Programming and Optimal Control," 2nd ed., Athena Scientific, Belmont, MA (2000)
Bertsekas DP, Tsitsiklis JN, "Neuro-Dynamic Programming," Athena Scientific, Belmont, MA (1996)
Chikkula Y, Lee JH, Ind. Eng. Chem. Res., 39(6), 2010 (2000)
Crites RH, Barto AG, "Improving Elevator Performance Using Reinforcement Learning," Advances in Neural Information Processing Systems 8, Touretzky, D.S., Mozer, M.C. and Haselmo, M.E., ed., MIT Press, Cambridge, MA, 1017 (1996)
Henson MA, Comput. Chem. Eng., 23(2), 187 (1998)
Howard RA, "Dynamic Programming and Markov Processes," MIT Press, Cambridge, MA (1960)
Kaisare NS, Lee JM, Lee JH, Int. J. Robust Nonlinear Control, 13, 347 (2002)
Lee JH, Cooley B, Chem. Process. Control, 5, 201 (1997)
Lee JH, Ricker NL, Ind. Eng. Chem. Res., 33(6), 1530 (1994)
Lee JH, Yu ZH, Automatica, 33(5), 763 (1997)
Lee JM, Lee JH, "Simulation-Based Dual Mode Controller for Nonlinear Processes," Proceedings of IFAC ADCHEM 2003, Accepted (2004)
Lee JM, Lee JH, "Neuro-Dynamic Programming Approach to Dual Control Problems," AIChE Annual Meeting, Reno, NV (2001)
Marbach P, Tsitsiklis JN, IEEE Trans. Autom. Control, 46(2), 191 (2001)
Mayne DQ, Rawlings JB, Rao CV, Scokaert POM, Automatica, 36(6), 789 (2000)
Meadows ES, Rawlings JB, "Model Predictive Control," Nonlinear Process Control, Henson, M.A. and Seborg, D.E., eds., Prentice Hall, New Jersey, 233 (1997)
Morari M, Lee JH, Comput. Chem. Eng., 23(4-5), 667 (1999)
Puterman ML, "Markov Decision Processes," Wiley, New York, NY (1994)
Sistu PB, Bequette BW, Chem. Eng. Sci., 50(6), 921 (1995)
Sutton RS, Barto AG, "Reinforcement Learning: An Introduction," MIT Press, Cambridge, MA (1998)
Tesauro GJ, Machine Learning, 8, 257 (1992)
VandeVusse JG, Chem. Eng. Sci., 19, 964 (1964)
Zhang W, Dietterich TG, "A Reinforcement Learning Approach to Job Shop Scheduling," Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, 1114 (1995)

[Referenced By]

[Related Articles]

[1] Astrom KJ, Helmersson A, Comp. Maths. Appl., 12A, 653 (1986)

[2] Astrom KJ, Wittenmark B, "Adaptive Control," Addison-Wesley (1989)

[3] Bellman RE, "Dynamic Programming," Princeton University Press, New Jersey (1957)

[4] Bemporad A, Morari M, Automatica, 35(3), 407 (1999)

[5] Bertsekas DP, "Dynamic Programming and Optimal Control," 2nd ed., Athena Scientific, Belmont, MA (2000)

[6] Bertsekas DP, Tsitsiklis JN, "Neuro-Dynamic Programming," Athena Scientific, Belmont, MA (1996)

[7] Chikkula Y, Lee JH, Ind. Eng. Chem. Res., 39(6), 2010 (2000)

[8] Crites RH, Barto AG, "Improving Elevator Performance Using Reinforcement Learning," Advances in Neural Information Processing Systems 8, Touretzky, D.S., Mozer, M.C. and Haselmo, M.E., ed., MIT Press, Cambridge, MA, 1017 (1996)

[9] Henson MA, Comput. Chem. Eng., 23(2), 187 (1998)

[10] Howard RA, "Dynamic Programming and Markov Processes," MIT Press, Cambridge, MA (1960)

[11] Kaisare NS, Lee JM, Lee JH, Int. J. Robust Nonlinear Control, 13, 347 (2002)

[12] Lee JH, Cooley B, Chem. Process. Control, 5, 201 (1997)

[13] Lee JH, Ricker NL, Ind. Eng. Chem. Res., 33(6), 1530 (1994)

[14] Lee JH, Yu ZH, Automatica, 33(5), 763 (1997)

[15] Lee JM, Lee JH, "Simulation-Based Dual Mode Controller for Nonlinear Processes," Proceedings of IFAC ADCHEM 2003, Accepted (2004)

[16] Lee JM, Lee JH, "Neuro-Dynamic Programming Approach to Dual Control Problems," AIChE Annual Meeting, Reno, NV (2001)

[17] Marbach P, Tsitsiklis JN, IEEE Trans. Autom. Control, 46(2), 191 (2001)

[18] Mayne DQ, Rawlings JB, Rao CV, Scokaert POM, Automatica, 36(6), 789 (2000)

[19] Meadows ES, Rawlings JB, "Model Predictive Control," Nonlinear Process Control, Henson, M.A. and Seborg, D.E., eds., Prentice Hall, New Jersey, 233 (1997)

[20] Morari M, Lee JH, Comput. Chem. Eng., 23(4-5), 667 (1999)

[21] Puterman ML, "Markov Decision Processes," Wiley, New York, NY (1994)

[22] Sistu PB, Bequette BW, Chem. Eng. Sci., 50(6), 921 (1995)

[23] Sutton RS, Barto AG, "Reinforcement Learning: An Introduction," MIT Press, Cambridge, MA (1998)

[24] Tesauro GJ, Machine Learning, 8, 257 (1992)

[25] VandeVusse JG, Chem. Eng. Sci., 19, 964 (1964)

[26] Zhang W, Dietterich TG, "A Reinforcement Learning Approach to Job Shop Scheduling," Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, 1114 (1995)