Styrene-butadiene-styrene (SBS) modified asphalt used in road engineering is a complicated mixture of various compositions. The present study was undertaken to evaluate the influence of different raw materials and their content on the moisture stability of SBS copolymer modified asphalt mixture. A total of 18 different combinations of base asphalt, SBS, furfural extract oil (FEO), and stabilizer were considered to prepare different SBS modified asphalt samples, and the contact angle test was carried out to determine the work of adhesion and the work of debonding of SBS modified asphalt with aggregate. Moreover, a multi-factor Analysis of Variance (ANOVA) was conducted to investigate the statistical effects of the material composition factors. The results show that the bond strength between SBS modified asphalt and aggregate is significantly affected by the source of base asphalt, but it is insensitive to the change of SBS type under dry and wet conditions. Besides, the moisture damage resistance of SBS modified asphalt improves firstly and then reduces with the rising storage of SBS, FEO, and stabilizer. Furthermore, linear regression analysis suggests that the work of adhesion under dry condition has excellent correlation with the work of debonding under wet condition.