A two-layer Lanczos algorithm is suggested to calculate the rovibrational energy levels of polyatomic molecules in terms of a partitioned Hamiltonian. Such a Hamiltonian is formed in a set of orthogonal polyspherical coordinates. This algorithm solves the full dimensional eigenvalue problem in a reduced-dimensional (RD) way. By splitting the coordinates into radial and angular groups, one obtains two small RD Hamiltonians in each coordinate group. The eigenstates of each RD system are computed using either a standard or a guided spectral transform Lanczos method. These two subsystems are exactly coupled via a set of diabatic basis functions in the angular degrees of freedom without any dynamical approximation. The two-layer Lanczos algorithm is illustrated in detail using an example of the variational calculation of the vibrational energies of pentatomic molecules. An application to methane is given. Numerical results show that the two-layer Lanczos method is substantially more efficient, compared to the conventional Lanczos algorithm.