With the growing use of renewable energy sources, the stability of electrical power systems can be seriously affected by fluctuations in the available power. As one of the potential solutions for this new challenge, the energy flexibility of buildings has become a focus for research and technological development. Most studies have focused on single buildings, with only a few studies on building clusters in which the building models were usually oversimplified in that they did not consider different building types or their thermal characteristics, their occupancy or their occupants' behaviour. In this paper, we describe a data driven approach to simulating a generic building cluster that could resemble any mix of building archetypes and occupancy. The energy flexibility potential of apartment building clusters was estimated by using data from surveys and available statistics in Denmark for the worst case scenario, i.e. when the end users do not allow any disturbance when they are at home, so that energy flexibility is only available when residents are not at home. In this scenario, no energy flexibility is assumed when buildings are occupied, which yields a conservative estimation. The uncertainty of the energy flexibility potential due to uncertain occupancy and various archetypes was quantified for different scales of building cluster. The resulting hybrid-model is a combination of a building model and an occupancy model and includes the different factors that influence the potential energy flexibility of buildings. The results show that the uncertainty of the energy flexibility decreases when the aggregated number of buildings increases. The uncertainty of energy flexibility was less than 10%, when about 700 households were aggregated. This approach can be used to simulate building energy flexibility for district or even regional level energy planning when the intention is to use the available flexibility to address the challenges caused by fluctuation in the power available from renewable energy sources.