Two quantum mechanical Hamiltonians have been derived in orthogonal polyspherical coordinates, which can be formed by Jacobi and/or Radau vectors etc., for the study of the vibrational spectra of six-atom molecules. The Hamiltonians are expressed in an explicit Hermitian form in the spatial representation. Their matrix representations are described in both full discrete variable representation (DVR) and mixed DVR/nondirect product finite basis representation (FBR) bases. The two-layer Lanczos iteration algorithm [H.-G. Yu, J. Chem. Phys. 117, 8190 (2002)] is employed to solve the eigenvalue problem of the system. A strategy regarding how to carry out the Hamiltonian-vector products for a high-dimensional problem is discussed. By exploiting the inversion symmetry of molecules, a unitary sequential 1D matrix-vector multiplication algorithm is proposed to perform the action of the Hamiltonian on the wavefunction in a symmetrically adapted DVR or FBR basis in the azimuthal angular variables. An application to the vibrational energy levels of the molecular hydrogen trimer (H-2)(3) in full dimension (12D) is presented. Results show that the rigid-H-2 approximation can underestimate the binding energy of the trimer by 27%. Finally, it is demonstrated that the two-layer Lanczos algorithm is also capable of computing the eigenvectors of the system with minor effort.(C) 2004 American Institute of Physics.