We present results of quantum mechanical scattering calculations on the vibrational relaxation of CO induced by collisions with He-3 at ultracold temperatures and compare them with the corresponding results for the He-4-CO system. The low energy cross sections are controlled by shape resonances supported by the van der Waals well. For a thermal rotational population the cross sections for the quenching of the v=1 vibrational level are larger for He-3 than for He-4 collisions. Because of the absence of a Feshbach resonance, the cross sections at low energies for the quenching of the j=1 rotational level are much smaller than those for He-4-CO. Very good agreement is found for the rate coefficients for the vibrational relaxation of the v=1 level with the measurement data that are available at temperatures above 35 K. There are small discrepancies between the calculated rate coefficients with earlier calculations for the relaxation of the v=1 vibrational level, similar to those found for He-4-CO collisions, but the discrepancies tend to be larger with decrease of temperature.