Asphaltene aggregation affects the entire production chain of the petrochemical industry, which also poses environmental challenges for oil pollution remediation. The aggregation process has been investigated for decades, but it remains unclear how the free energy of asphaltene association in solvents is correlated to its molecular structure. In this study, dimerization energies of 28 types of asphaltenes in water and toluene were calculated using the umbrella sampling method. Structural parameters related to the atom types and functional groups were screened to identify the factors most influencing the dimerization energy using multiple linear regression, multilayer perceptron, and support vector regression. Results demonstrated that the influence of molecular structure on asphaltene association in water was nonlinear, while attempts to capture the relationship using linear regression had larger error. The linkage per aromatic ring, number of aromatic carbons, and number of aliphatic chains were the top three factors accounting for 52% of the dimerization energy variation in water. Asphaltene dimerization in toluene was dominated by the content of sulfur in aromatic rings and the number of aromatic carbons which contributed to 55% of the energy variation. To the best of our knowledge, this was the first study successfully predicting asphaltene dimerization using molecular structure (R > 0.9) and quantifying simultaneously the relative importance of each structural parameter. The proposed modeling approach supported the decision making on the number of structural parameters to investigate for predicting asphaltene aggregation.