We investigate the influence of the microstructure on the fracture properties of calcium aluminate cement/polymer composites. We carry out microscopic scratch tests during which a Rockwell C diamond probe pushes across the surface of a polished specimen under a linearly increasing vertical force. We extend the scratch fracture method to heterogeneous materials. The scratch test induces a ductile-to-brittle transition as the penetration depth increases. Scanning electron microscopy imaging shows that the low porosity and the strong cement-binder interphase favor toughening mechanisms such as crack trapping and bridging. Nonlinear fracture mechanics theory yields the fracture toughness in the fracture-driven regime. The fracture toughness of macro-defect-free (MDF) cement is found to decrease as the polymer-to-cement ratio increases. This decrease in the fracture resistance can be explained by the decrease in anhydrous cement content and the increase in the inter-particle distance between cement grains. By evaluating the fracture toughness of the micro-constituents of MDF cement, we show that the high value of the fracture toughness at the composite level stems from tough calcium aluminate phases and a highly packed non-porous granular microstructure.