A new kind of self-dispersing silica nanoparticle was prepared and used to enhance oil recovery in spontaneous imbibition tests of low-permeability cores. To avoid the aggregation of silica nanoparticles, a new kind of silica nanoparticle was prepared through the surface modification with vinyltriethoxysilane and 2-mercaptobenzimidazole as modified agents. Transmission electron microscopy, Fourier transform infrared spectroscopy, dynamic light scattering, and potential measurements were employed to characterize the modified silica nanoparticles. Dispersing experiments indicated that modified silica nanoparticles had superior dispersity and stability in alkaline water. To evaluate the performance of silica nanofluids for enhanced oil recovery compared to pH 10 alkaline water and 5 wt % NaCl solution, spontaneous imbibition tests in sandstone cores were conducted. The results indicated that silica nanofluids can evidently improve oil recovery. To investigate the mechanism of nanoparticles for enhanced oil recovery, the contact angle and interfacial tension were measured. The results showed that the adsorption of silica nanoparticles can change the surface wettability from oil-wet to water-wet and silica nanoparticles showed a little influence on oil/water interfacial tension. In addition, the change of the oil droplet shape on the hydrophobic surface was monitored through dynamic contact angle measurement. It was shown that silica nanoparticles can gradually detach the oil droplet from the hydrophobic surface, which is consistent with the structural disjoining pressure mechanism.