The material removal mechanism of ultrasonic machining sintered Nd-Fe-B magnetic materials was studied theoretically. The relation between critical load and central crack is given. In order to assure the material removal mode on material surface is brittle micro-fracture, the acting force of a single abrasive particle working on workpiece surface should be higher than the critical load. Experimental results show that there should be an optimal static load and an optimal abrasive size in certain ultrasonic machining system. The research results are helpful to guide practical production.