The theory of nonideal microwave pulses acting on electron-nuclear spin systems is extended and applied to optimize the two-pulse electron spin echo envelope modulation (ESEEM) experiment. A superoperator approach for a computationally efficient simulation of experiments involving non-ideal pulses is introduced and the corresponding unitary transformation superoperator is given analytically for a system consisting of one electron spin S=1/2 and one nuclear spin I=1/2. Density operator single-element transfers are divided into allowed and forbidden ones and are classified according to their functioning in pulse ESR. By increasing the efficiency of forbidden transfers by Hartmann-Hahn matching during prolonged pulses, the sensitivity of the conventional two-pulse ESEEM experiment may drastically be improved and discrimination between basic, hyperfine, and combination frequencies becomes possible. The implications of the theory for spin systems with an arbitrary number of nuclear spins 1/2 are investigated by deriving and discussing a general condition for Hartmann-Hahn matching of forbidden transitions. It is shown that the product rule valid for two-pulse echo modulations caused by more than one nucleus does not hold for nonideal pulses. A method is developed that allows one to reduce the thus arising large dimensionality of the diagonalization problem in numerical simulations. The theoretical conclusions are verified by experiments on two transition metal complexes in single crystals and on a spin-label-doped polymer sample.