The maximum capacity, the amount of maximal electric charge that a battery can store, not only indicates the state of health, but also is required in numerous methods for state-of-charge estimation. This paper proposes an alternative approach to perform online estimation of the maximum capacity by solving the recursive total-least-squares (RTLS) problem. Different from prior art, the proposed approach poses and solves the RTLS as a Rayleigh quotient optimization problem. The Rayleigh quotient-based approach can be readily generalized to other parameter estimation problems including impedance estimation. Compared with other capacity estimation methods, the proposed algorithm enjoys the advantages of existing RTLS-based algorithms for instance, low computation, simple implementation, and high accuracy, and thus is suitable for use in real-time embedded battery management systems. The proposed method is compared with existing methods via simulations and experiments.