Doppler-free two-photon fluorescence excitation spectra of the (A) over tilde (1)A(u)(v(7)'=1)<--<(X)over tilde> (1)A(g)(v"=0) transition of trans-glyoxal were measured by propagating a laser beam either polarized parallel (pi pump) or perpendicular (sigma pump) to the magnetic field of 6 T. pi pump and sigma pump dependence of the Zeeman spectra of the (Q(K))Q(J=6) lines for K=0-6 was measured. Intensities of the Zeeman components of the (Q(6))Q(6) lines were observed to have the maxima at the high and low wave number ends and the minimum at the middle for pi pump, and the minima at the high and low wave number ends and the maximum at the middle for sigma pump. By comparing calculated and observed patterns of the Zeeman spectra, it became clear that the transition tensor M-00 is dominant and the effective intermediate state is 1 B-1(u)(pipi(*)). K dependence of the Zeeman spectra of the (Q(K))Q(J=15) lines was measured, and the Zeeman splittings were observed to increase proportionally to K-2. J dependence of the Zeeman spectra of the (Q(K=J))Q(J) lines was measured for J=0-15, and the Zeeman splittings was found to increase proportionally to J. The K and J dependence of the Zeeman splittings were analyzed. It is concluded that the Zeeman splitting of the (A) over tilde (1)A(u)(npi(*)) state is originating from the mixing of the 1 B-3(u)(pipi(*)) state by spin-orbit interaction, and the magnetic moment is along the molecular top axis. (C) 2003 American Institute of Physics.