The onset of convection in a horizontal layer of a porous medium saturated with a viscoelastic nanofluid was studied in this article. The modified Darcy model was applied to simulate the momentum equation in porous media. An Oldroyd-B type constitutive equation was used to describe the rheological behavior of viscoelastic nanofluids. The model used for the viscoelastic nanofluid incorporates the effects of Brownian motion and thermophoresis. The onset criterion for stationary and oscillatory convection was analytically derived. The effects of the concentration Rayleigh number, Prandtl number, Lewis number, capacity ratio, relaxation, and retardation parameters on the stability of the system were investigated. Oscillatory instability is possible in both bottom- and top-heavy nanoparticle distributions. Results indicated that there is competition among the processes of thermophoresis, Brownian diffusion, and viscoelasticity that causes the convection to set in through oscillatory rather than stationary modes. Regimes of stationary and oscillatory convection for various parameters were derived and are discussed in detail.