Experimental results on the hydrodynamic stability of three dilute polymer solutions (polyisobutylene-based Boger fluids) in Taylor-Couette flow are presented. We demonstrate that viscous heating causes significant destabilization relative to the isothermal viscoelastic case, and we quantify the effect of viscous heating on the critical conditions. Experiments in which the shear rate was stepped to a steady value indicate that the onset time for instability scales with the product of the polymeric relaxation time and the time for the fluid to reach a steady temperature profile under the influence of viscous heating. This time scale was used to perform quasistatic ramp tests to determine the critical Deborah number (De) as a function of the Nahme number, or the magnitude of viscous heating. At low Nahme and Peclet numbers, the critical De remains essentially constant and the disturbance flow is nonaxisymmetric and oscillatory. As the Nahme number (Na) and Peclet number (Pe) are increased, the disturbance flow changes to an axisymmetric and stationary vortex flow similar to Taylor vortices, and the critical De decreases dramatically. At Pe approximate to 108 000 and Na approximate to 0.1, corresponding to a maximum temperature increase in the gap of 1.6 K, the critical Deborah number decreases to a value that is 12% of the plateau value of De, observed at low Na and Pe. These observations are in agreement with linear stability predictions by Al-Mubaiyedh and co-workers (1999, 2000) of viscoelastic Taylor-Couette flow for a multimode Oldroyd-B fluid in which viscous heating is present. (C) 2003 The Society of Rheology.