Novel self-cleaning PES nanofiltration membranes were successfully synthesized via the phase inversion method with different amounts of the boron doped-TiO2-SiO2 cobalt ferrite (B-TiO2-SiO2/CoFe2O4) nanophotocatalyst. The nanofiltration performance was investigated by the rejection of direct red 16 and COD removal of the biologically treated palm oil mill effluent (POME). The addition of the B-TiO2-SiO2/CoFe2O4 nanoparticles into the polymeric casting solution improved porosity, morphology, structure, pure water flux (PWF), antifouling properties and separation performance of the embedded membranes due to hydrophilic behavior and photocatalytic properties of the B-TiO2-SiO2/CoFe2O4 nanoparticles. The B-TiO2-SiO2/CoFe2O4 nanoparticles can degrade adsorbed organic foulants on the membrane surface (reversible fouling) or deposited/trapped fouling agents into the membrane pores (irreversible fouling). The prevention of the interaction between organic foulant and hydrophilic membrane improves antifouling properties and wettability of the modified membranes. The membrane with 0.5 wt% of the B-TiO2-SiO2/CoFe2O4 nanoparticle had the highest pure water flux and flux recovery ratio (FRR) with the best separation performance. However, at high concentration of nanoparticles, agglomeration of the nanoparticles on membrane surface/pores resulted in pore blocking, roughness membrane surface which led to slightly reduction of PWF, FRR and separation performance. The obtained results were also confirmed by contact angle measurement, scanning electron microscopy (SEM) and atomic force microscopy analysis (AFM).