Controlling the surface roughness of thin films with nanoscale precision is of significant interest for the rational design of surface coatings. Although wrinkling and buckling of Langmuir monolayers under compression has been demonstrated for several years, there is currently no method to precisely control this behavior during compression and thereby modify the surface roughness of deposited films. Here, we combine conventional Langmuir phase analysis with a novel dynamic viscoelasticity measurement to simply and accurately observe the jamming transition of monolayers of silica spheres, graphene oxide, and surfactant. By overcompressing beyond this point, the surface roughness of the deposited monolayer can be precisely controlled. This technique could be used to tune the surface properties of a variety of materials from lipids to nanoparticles.