A negative characteristic of low-emission flames is their weak dynamic flame stability, which leads in many cases to periodic pulsations of heat release and combustor pressure. Due to the extremely complex nature of the forcing and feedback mechanisms in flows with heat release, reliable prediction of the stability limits of combustors has not been achieved yet with satisfactory accuracy. The dynamics of complicated thermoacoustic systems, which are modeled on the basis of linear acoustics and a representation of the system as a network of acoustic elements, can be analyzed using methods derived from control theory. It was shown in a companion article (Sattelmayer and Polifke, Combustion Science and Technology, vol. 175, pp. 453 476, 2003) that the commonly employed open-loop Bode plot stability analysis can lead to erroneous results. Similar problems may occur also for a Nyquist diagram, if the interpretation of the open-loop gain curve follows the standard rules of traditional control theory. A lack of suitable methods is clearly visible. In this work, a generalized formulation of the Nyquist stability criterion is proposed that takes the special character of thermoacoustic systems into account. It is found that the new formulation leads to major improvements. All unstable modes of a model system are detected properly. Furthermore, the mathematical background of the new rules for the interpretation of the open-loop gain form the basis for a proposed extension of the Nyquist method that delivers the growth rate of eigenmodes as well as their frequencies. From the viewpoint of technical applications, it is important that this novel method can be easily applied to networks that incorporate elements for which a closed-form analytical expression for the transfer matrix is not known. Important examples for such elements are transfer matrices of burners or flames obtained from computational fluid dynamics or from measurement. For this reason, the method is more generally applicable to complex, applied thermoacoustic systems than to the direct determination of the complex eigenfrequencies.