Instability triggering and transient growth of thermoacoustic oscillations were experimentally investigated in combination with linear/nonlinear flame transfer function (FTF) methodology in a model lean-premixed gas turbine combustor operated with CH4 and air at atmospheric pressure. A fully premixed flame with 10 kW thermal power and an equivalence ratio of 0.60 was chosen for detailed characterization of the nonlinear transient behaviors. Flame transfer functions were experimentally determined by simultaneous measurements of inlet velocity fluctuations and heat release rate oscillations using a constant temperature anemometer and OH*/CH* chemiluminescence emissions, respectively. The phase-resolved variation of the local flame structure at a limit cycle was measured by planar laser-induced fluorescence of OH. Simultaneous measurements of inlet velocity, OH*/CH* emission, and acoustic pressure were performed to investigate the temporal evolution of the system from a stable to a limit cycle operation. This measurement allows us to describe an unsteady instability triggering event in terms of several distinct stages: (i) initiation of a small perturbation, (ii) exponential amplification, (iii) saturation, (iv) nonlinear evolution of the perturbations towards a new unstable periodic state, (v) quasi-steady low-amplitude periodic oscillation, and (vi) fully-developed high-amplitude limit cycle oscillation. Phase-plane portraits of instantaneous inlet velocity and heat release rate clearly show the presence of two different attractors. Depending on its initial position in phase space at infinitesimally small amplitude, the system evolves towards either a high-amplitude oscillatory state or a low-amplitude oscillatory state. This transient phenomenon was analyzed using frequency- and amplitude-dependent damping mechanisms, and compared to subcritical and supercritical bifurcation theories. The results presented in this paper experimentally demonstrate the hypothesis proposed by Preetham et al. based on analytical and computational solutions of the nonlinear G-equation [J. Propul. Power 24 (2008) 1390-1402]. Good quantitative agreement was obtained between measurements and predictions in terms of the conditions for the onset of triggering and the amplitude of triggered combustion instabilities. (C) 2011 The Combustion Institute. Published by Elsevier Inc. All rights reserved.