We consider torsional flow of a viscoelastic fluid in a cone-and-plate device. This flow is known to undergo a purely elastic instability when the Deborah number reaches a critical value. Beyond this critical value a Hopf bifurcation to spiral vortices occurs. In this paper we consider the stability of the flow to non-axisymmetric disturbances when the Reynolds number is non-zero. We examine the effect of inertia on the critical value of the Deborah number at the onset of instability, the winding number of the spiral waves, as well as the wave number of the vortices. The constitutive model of Oldroyd-B is used in the present analysis. Our results show that in general when the cone angle is small the stability characteristics of the flow do not change much with inertia, indicating that the creeping model is indeed a very good approximation in such cases. We show that the critical Deborah number tends to increase with inertia in the case of non-axisymmeric disturbances. One important implication of our results is that whereas the creeping flow approximation gives a good prediction of the onset of instability the post critical bifurcations will be influenced by the inertial terms. In particular, since inertia tends to stabilize non-axisymmetric modes while destabilizing axisymmetric modes, the interaction of the two modes could be more significant than is predicted by the creeping flow results. Indeed, experimental results reported in McKinley et al., 1995, J. Fluid. Mech. 285, 123, show that for parameter values for which the creeping flow equations predict bifurcations to spiral vortices, purely axisymmetric modes were also observed. An energy analysis of the non-axisymmetric modes shows the mechanism driving the instability to be the coupling between the perturbation polymeric stress and the base velocity.