Conjugated polymers are organic semiconductors that have emerged over the past few decades as promising alternatives to inorganic semiconductors for use in a range of electronic applications. Since the performance of these materials in various devices is dependent on many variables, there have been numerous reports on the design and synthesis of novel materials. In this Perspective, we focus specifically on the use of heteroatom substitution to tune the properties of conjugated polymers. This strategy is very promising as the subtle modification in the polymers structure can often result in significant changes in the optical and electronic properties of the materials. Given the current interest in renewable energy, we are highlighting the impact these modifications have on the performance of these conjugated polymers in photovoltaic cells. Based on the popularity of thiophene-based conjugated polymers, the focus of this work is the group 16 elements oxygen, sulfur, selenium, and tellurium. In recent years, we have seen large number of papers utilizing this approach with mixed results. In addition to summarizing the work in this area, we will also identify opportunities and challenges