Real world mechanical systems present nonlinear behavior and in many cases simple linearization in modeling the system would not lead to satisfactory results. Coulomb damping and cubic stiffness are typical examples of system parameters currently used in nonlinear models of mechanical systems. This paper uses orthogonal functions to represent input and output signals. These functions are easily integrated by using a so-called operational matrix of integration. Consequently, it is possible to transform the nonlinear differential equations of motion into algebraic equations. After mathematical manipulation the unknown linear and nonlinear parameters are determined. Numerical simulations, involving single and two degree-of-freedom mechanical systems, confirm the efficiency of the above methodology.