A two-dimensional numerical model is presented to examine meso- and macroscopic structure effects on the effective thermal conductivity of concrete. The heterogeneity of concrete is considered at a mesoscopic level by Weibull distribution assumption. Simulations on several heterogeneous samples show that the effective thermal conductivity strongly depends on the degree of heterogeneity. Higher homogeneity indicates a greater effect on the effective thermal conductivity. Numerically simulated results also indicate that the size and shape of individual coarse aggregate appear to have negligible influence on the effective thermal conductivity of concrete. However, that greatly depends on the thermal conductivity and volume fraction of coarse aggregate. Modeling suggests that heat conductivity decreases when there is a drop in strength due to the damages creating a thermal barrier across the cracks and thus preventing heat flow through the matrix, that is, resulting in reduction of effective thermal conductivity. Moreover, the formation of cracks in the interfacial transition zone also leads to significant reduction of heat flow in coarse aggregates.