This study investigates the performance of a wide selection of single objective black-box optimization algorithms (optimizers) when applied to a large set of building energy simulation problems from the literature. Optimizers include randomized, deterministic and model-based algorithms. We also study the impact of tunable hyper-parameters on performance for some of the optimizers. We apply benchmark metrics from the literature and develop a new metric based on ranks, which is especially useful for comparing small differences in optimal cost values. Our extensive experimental setup allows us to draw generalized conclusions on convergence times, stability and robustness of the optimizers in the context of building energy simulations. Main findings include the importance of tuning optimization algorithm parameters, consistently very fast convergence with the model-based optimizer RBFOpt but with a risk of being trapped in local optima, competitive results with classical metaheuristics such as a simple Genetic Algorithm and Particle Swarm Optimization Algorithm when tuned to specific problem dimensions, and consistently good results with CMA-ES when given a very large evaluation budget. It is striking however that no single algorithm dominates the benchmark for all considered performance metrics and/or optimization problems. Another important finding is that many of the discrete design variables in building energy optimization are in fact ordinal, which makes algorithm operators designed for continuous variables more effective than operators designed for combinatorial problems. These findings help users to select an appropriate optimizer for BEO problems and the available evaluation budget. Finally, this study provides an open-source framework for other researchers to benchmark their algorithms on the same test bed, hence enabling traceable and more rigorous performance comparisons. (C) 2019 Elsevier B.V. All rights reserved.