Natural ventilation is a green building strategy that improves building energy efficiency, indoor thermal environment, and air quality. However, in practice, it is not always clear when and how to utilize the natural ventilation and coordinate its operation with the HVAC system. This paper introduces a reinforcement learning control strategy, specifically through model-free Q-learning, that makes optimal control decisions for HVAC and window systems to minimize both energy consumption and thermal discomfort. This control system evaluates the outdoor and indoor environments (temperature, humidity, solar radiation, and wind speed) at each time step, and responds with the best control decision that targets both immediate and long-term goals. The reinforcement learning control is evaluated through numerical simulation on a building thermal model and compared with a rule-based heuristic control strategy. Case studies in hot-and-humid Miami and warm-and-mild Los Angeles demonstrated the superior performance of reinforcement learning control, which led to 13% and 23% lower HVAC system energy consumption, 62% and 80% lower discomfort degree hours, and 63% and 77% fewer high humidity hours compared to heuristic control. Unlike heuristic control that requires specific knowledge of individual buildings and the creation of exhaustive decision-making scenarios to improve performance, reinforcement learning control guarantees optimality through self-advancement on given goals and cost functions and is able to adapt to stochastic occupancy and occupant behaviors, which is difficult to accommodate by heuristic control. (C) 2018 Elsevier B.V. All rights reserved.