Most quality control pathways target misfolded proteins to prevent toxic aggregation and neurodegeneration(1). Dimerization quality control further improves proteostasis by eliminating complexes of aberrant composition(2), but how it detects incorrect subunits remains unknown. Here we provide structural insight into target selection by SCF-FBXL17, a dimerization-quality-control E3 ligase that ubiquitylates and helps to degrade inactive heterodimers of BTB proteins while sparing functional homodimers. We find that SCF-FBXL17 disrupts aberrant BTB dimers that fail to stabilize an intermolecular beta-sheet around a highly divergent beta-strand of the BTB domain. Complex dissociation allows SCF-FBXL17 to wrap around a single BTB domain, resulting in robust ubiquitylation. SCF-FBXL17 therefore probes both shape and complementarity of BTB domains, a mechanism that is well suited to establish quality control of complex composition for recurrent interaction modules. Structural studies of the dimerization quality control E3 ubiquitin ligase SCF-FBXL17 indicate that its selectivity for aberrant complex formation is based on recognizing both shape and complementarity of interacting domains.