The phase behavior of mixtures of magnetic colloids and a nonadsorbing polymer has been investigated using mean field theory. The polymer in this mixture causes an effective isotropic attraction (depletion attraction) between the magnetic colloids, the range and strength of which can be varied independently from the (anisotropic) dipole-dipole interaction. Calculations within this mean field approximation show that in the absence of a magnetic field a magnetic colloid-polymer mixture may phase separate into a polymer-rich colloidal gas and a polymer-poor colloidal liquid. Upon application of a magnetic field, the minimum amount of polymer needed to destabilize the mixture is reduced. However, for a dipole-dipole interaction strength typical for real ferrofluids, this reduction is insufficient to give phase coexistence in a polymer-free ferrofluid. The theory also suggests that free oleic acid, which is present in many ferrofluids, may decrease a ferrofluid's stability, in the same way that a polymer does.