We study the aggregation kinetics of two elastomer colloids with similar bulk polymer properties but with different surface charge groups in order to understand the role of the surface properties in particle coalescence during aggregation. It is confirmed that clusters of the elastomer particles stabilized purely by ionic surfactants coalesce in both reaction-limited and diffusion-limited aggregation (RLCA and DLCA) regimes and that the coalescence is independent of the coagulant type. On the other hand, clusters formed by elastomer particles stabilized by charged polymer end groups, which are fixed on the particle surface, are fractal objects with a fractal dimension of 1.7 in the DLCA and 2.1 in the RLCA regime. This indicates insignificant cluster coalescence during aggregation, most likely due to a hindrance effect or the fixed charges.