Manufacture of ultra large-scale integrated circuits involves accurate control of a challenging nonlinear Rapid Thermal Processing (RTP) system. Precise control of temperature profile and rapid ramp-up and ramp-down rates demanded by a RTP system cannot be achieved with conventional control strategies due to nonlinear and multi time-scale effects. In this paper the control of a RTP system is reformulated as an optimal multi-step sequential decision problem using the framework of finite horizon Markov decision processes and solved using a Reinforcement Learning (RL) algorithm. Three increasingly complex RL based control strategies are explored and compared with the existing state-of-the-art approach for controlling RTPs. Simulation results indicate that the approach proposed in this paper achieves superior control of the temperature profile and ramp-up and ramp-down rates for the RTP system. (C) 2018 Elsevier Ltd. All rights reserved.