A novel and efficient approach for modeling irregular domain boundaries and enforcing complex landbased constraints in wind farm layout optimization is presented. The presented method can be used to enforce constraints and represent complex domains for which an analytical expression is not available. The developed approach uses support vector domain description to obtain an analytical expression of the domain from data points sampled from the various regions for wind turbine placements. Such an analytical, continuous, differentiable description can then be used for wind farm layout optimization using gradient based and non-gradient based optimization algorithms. The applicability of the method was demonstrated on a wind farm layout optimization problem where several regions of the terrain were restricted. The method was shown to accurately and efficiently model regular, irregular, convex, nonconvex and discontinuous domains/regions using only scattered data sampled from the domains. An efficient method for terrain modeling and interpolation is also presented. The proposed interpolation method uses quadtree decomposition to decompose the large interpolation problem into several smaller interpolation problems thereby reducing the complexity of the interpolation. The framework developed (libSVDD) is also made publicly available. (C) 2020 Elsevier Ltd. All rights reserved.