An unsaturated ladder phenyl/vinyl polysilsesquioxane (PhVPOSS) was synthesized and applied to vinyl ester resin (VER) thermosets. The compatibility, mechanical properties, thermal properties, and combustion behavior of the VER-PhVPOSS composites were further investigated. As a result, the PhVPOSS and VER had a good compatibility and interaction based on the solubility parameters and Flory-Huggins parameters. PhVPOSS provided the VER with a significantly enhanced bending strength, thermal stability, and smoke-suppression effect. When 20 wt% PhVPOSS was added, the bending strength, flexural modulus, and breaking force increased by195.6%, 260.6%, and 196.6% compared with pure VER. The peak heat release rate, total smoke release, and specific optical density decreased by 52.2%, 20.7%, and 15.7%, respectively. These favorable characteristics indicated that PhVPOSS had a good application prospect in unsaturated polyester resin. [GRAPHICS] .