Obtaining complex topological micro- and nanomaterials in a controlled way is an open challenge for material science and chemistry. Recent experimental and computational studies have demonstrated the feasibility of self-assembling knots with up to eight crossings starting from small, identical building blocks. In this work, we investigate computationally a different pathway for knot production. By performing extensive computer simulations of hetero patchy polymers with different patch geometries, we show that it is possible to obtain both torus and twist knots, up to knots with more than 12 crossings. Our results indicate that with patchy polymers it is possible to exploit the bending rigidity of the backbone, the specific geometry of the patches, and the alphabet size to control the spectra of knots of the polymer. In particular, we find that increasing the alphabets to 20 letters tends to suppress knots, a finding that points to a new hypothesis to explain the rarity of knots in proteins. Finally, we demonstrate the ability to fold specific knotted conformations with high precision by designing the heteropolymer sequence. These include both diffuse and highly localized knots as well as two topologies which have not yet been synthesized by self-assembly: the 5(2) and 10(124) knots.