Usually, the fatigue fracture of ordinary aluminum-silicon alloys has been ascribed to the existence of surface or sub-surface pores. However, the relationships between various microstructural parameters and crack propagation behavior should be evaluated in terms of effects on the overall lifetime. In such a sense, the reported interpretations have been so simple and straightforward to discern the complicated interactions among various parameters. In this study, initiation and evolution of damage at eutectic and primary Si particles during monotonic and cyclic loadings are investigated utilizing in-situ studies. Al-7%Si and Al-20%Si binary alloys are produced as model materials. The damaging behaviors are analyzed in terms of the composite theory. In the tensile tests, particles are found to crack perpendicularly to a loading axis, being triggered by a far-field uniaxial stress. The damage is accumulated gradually with the increase in applied strain. Estimated in-situ strengths of the Si particles are 500 to 900MPa for the small eutectic Si particles, whereas it is as low as 200MPa for the coarse primary Si particles. During the cyclic loading, gradual accumulation of the damage at the Si particles is observed with the increase in number of cycles. Propagating through such a weakened zone, a fatigue crack seems to receive some acceleration due to the amplification mechanisms caused by the existence of the cracked particles ahead of it. The degree of the acceleration is quantitatively evaluated by experiments.