The two-layer Lanczos algorithm has been extended to study the rovibrational spectrum of four-atom molecules in a mixed grid/finite basis representation basis. This algorithm exploits the partitioning structure of the Hamiltonian in polyspherical coordinates, and calculates the eigenstates in a reduced-dimension (RD) way but exactly. It consists of the standard Lanczos method for the outer layer iteration and the guided spectral transform (GST) Lanczos for the inner layer. For H2CO, numerical results show that the two-layer Lanczos method is substantially efficient. Published by Elsevier Science B.V.