Colloidal monolayers are important tools to fabricate surface structures at the nanoscale. A typical monolayer fabrication strategy involves the self-assembly of colloidal building blocks at liquid interfaces, which are subsequently deposited on a solid substrate. Even though this process is well established, the resulting order of the particles within the colloidal monolayer differs between batches of colloidal particles and can even change with the age of the dispersion. In this study, we investigate the origins of this variation of monolayer quality for polystyrene particles synthesized by surfactant-free emulsion polymerization. We correlate the interfacial behavior of the colloidal particles at the air/water interface on a Langmuir trough with the resulting quality of the monolayer after transfer to a solid substrate. We identify surface-active impurities as a major cause for a disturbed self-assembly of the colloidal particles. These impurities form during the particle synthesis and consist of copolymers of styrene, the comonomer acrylic acid, and sulfonate species from the initiator. We show that they can be removed by cleaning protocols to increase the monolayer quality. However, our experiments demonstrate that the impurities reappear over time even for cleaned dispersions, indicating desorption from the surface of the colloidal particles. We identify strategies to avoid the presence of the impurities at the air/water interface or to inhibit their effect on the self-assembly process. These simple guidelines improve the quality of the resulting colloidal monolayer, which is a prerequisite for the reliable fabrication of high-quality surface nanostructures from colloidal templates.