Antifouling coatings have attracted much attention in applications of industrial equipment and systems. The accumulation of foulants on downhole equipment used in the steam-assisted gravity drainage process for oil sand extraction reduces the recovery efficiency of bitumen and causes failures of the equipment. Applying suitable coatings is expected to endow antifouling property to the related substrate surface. Understanding the interaction mechanism between foulants and antifouling coating is critical for the evaluation and prediction of antifouling performance. In this work, antifouling coating consisting of Ni and P was prepared on the L80 carbon steel substrate, which is a commonly used material in downhole equipment. The surface properties (e.g., morphology and surface energy) of the Ni-P coating and pristine L80 substrates were characterized. The intermolecular and surface forces between typical foulants (e.g., silica and bitumen-coated silica) and Ni-P coated and uncoated L80 substrates were directly measured using the atomic force microscope colloidal probe technique to investigate the fouling mechanisms and predict the fouling behaviors of the foulants at the nanoscale. It was found that the Ni-P coating possessed lower surface energy, weaker attractive van der Waals interaction, and smaller adhesion to both silica and bitumen as compared to the uncoated L80 substrate. Bulk soaking experiments were also conducted accordingly which further demonstrated the antifouling performance of Ni-P coating against silica and bitumen, agreeing well with the surface force measurements. This work has improved the fundamental understanding of fouling behaviors of silica and bitumen and the antifouling mechanism of Ni-P coating at the nanoscale, with potential applications in many related processes in chemical and petroleum industries. The methodologies employed in this work can be readily extended for investigating fouling and antifouling mechanisms of different materials in various engineering fields.