In order to improve the intrinsically brittle nature of the oil-well cement stone, a new type of submicron-fibrous whisker, referred to as the silicon carbide whisker (SiC whisker) was added into cement paste. The cement-based composites were prepared with the various amounts of submicron SiC whiskers. The composites were analyzed via scanning electron microscopy (SEM), mechanical testing, X-ray diffraction (XRD), infrared spectrum analysis (FTIR), and thermal stability analysis in order to understand the enhanced effect that the submicron SiC whiskers had on the applied properties and mechanisms of oil-well cement stone. The compressive strength, the tensile strength, and the flexural strength of the sample with 1% of additional submicron SiC whiskers increased by 9%, 104%, and 26%. The ultimate strain was more than 5%, which increased 194%. The elasticity modulus was 4648.4MPa, which decreased 31.1% after 28 curing days. These improved the mechanical properties of the cement-based composites primarily because of the microscopic mechanism of reinforcement via the submicron SiC whisker, including the bridging effect, the crack deflection, and the pull-out effect.