This paper presents a new nonlinear algorithm of speed regulation system for a flux-modulated permanent magnet in-wheel (FMPMW) motor. To improve the robustness and static performance of speed regulation system, sliding mode control with a sigmoid function is adopted. Compared with PI controller, the proposed algorithm can improve the robustness and minimize the torque ripple of the system. In addition, an adaptive notch filter is proposed to restrain the fifth and seventh current harmonics, which the static performance of control system can be further improved. Based on a 2-kW FMPMW prototype motor, the simulation and experimental results verify the good robustness and static performance of the proposed algorithm.