The giant optical nonlinearity associated with photoinduced molecular reorientation of dye-doped nematic liquid crystals has been measured for a homologous set of dyes belonging to the anthraquinone family, dissolved both in a polar and in a nonpolar liquid crystal host. We found a strong sensitivity of the nonlinearity merit figure to the detailed structure of the dye substituent groups. Our results provide some insight into the molecular mechanisms underlying this phenomenon, whose full understanding will finally require a detailed picture of guest-host intermolecular interactions and their dependence on the molecule electronic state.