In order to study the mechanical properties and the progressive failure process of composite under shear loading, a representative volume element (RVE) of fiber random distribution was established, with two dominant damage mechanisms - matrix plastic deformation and interfacial debonding - included in the simulation by the extended Drucker-Prager model and cohesive zone model, respectively. Also, a temperature-dependent RVE has been set up to analyze the influence of thermal residual stress. The simulation results clearly reveal the damage process of the composites and the interactions of different damage mechanisms. It can be concluded that the in-plane shear fracture initiates as interfacial debonding and evolves as a result of interactions between interfacial debonding and matrix plastic deformation. The progressive damage process and final failure mode of in-plane shear model which are based on constitute are very consistent with the observed result under scanning electron microscopy of V-notched rail shear test. Also, a transverse shear model was established as contrast in order to comprehensively understand the mechanical properties of composite materials under shear loading, and the progressive damage process and final failure mode of composite under transverse shear loading were researched. Thermal residual stress changes the damage initiation locations and damage evolution path and causes significant decreases in the strength and fracture strain.