The oscillatory flow instability in a liquid metal cylinder with a free upper surface, exposed to a rotating magnetic field (RMF), is analyzed by numerical simulations of the axisymmetric Navier-Stokes equations. The critical Taylor number designating the onset of the oscillatory flow regime is lower than that for the development of Taylor-Gortler vortices and decreases with increasing aspect ratio A = H-0/2R(0). In parallel, the Taylor-number interval, where the flow oscillations occur, becomes narrower. The instability is initiated near the free surface, where an oscillatory variation of both the size and the position of the upper vortex in the secondary flow can be observed accompanied by horizontal oscillations of the azimuthal velocity maximum at the free surface. The predicted flow regime has been observed in corresponding model experiments with GaInSn using Ultrasound Doppler Velocimetry (UDV) for flow field measurements. The occurrence of the oscillatory flow regime depends sensitively on the cleanliness of the liquid metal surface. (C) 2011 Elsevier B.V. All rights reserved.