Rheum officinale Baill., an important but endangered medicinal herb, is endemic to China. Inter-simple sequence repeat (ISSR) markers were employed to investigate the genetic diversity and differentiation of 12 populations of R. officinale. Thirteen selected primers yielded 189 bright and discernible bands, with an average of 14.54 per primer. The genetic diversity was low at the population level, but pretty high at the species level (H = 0.1008, I = 0.1505, PPB = 28.95% vs. H = 0.3341, I = 0.5000, PPB = 95.24%, respectively) by POPGENE analysis. Analysis of molecular variance (AMOVA) showed that the genetic variation was found mainly among populations (74.38%), in line with the limited gene flow (N-m = 0.2766) among populations. Mantel test revealed a significant correlation between genetic and geographic distances (r = 0.5381, P = 0.002), indicating the role of geographic isolation in shaping the present population genetic structure. Both Bayesian analysis and UPGMA cluster analysis demonstrated the similar results. Our results imply that the conservation efforts should aim to preserve all the extant populations of this endangered species, and cultivation is proposed in this study.