In water, cloud points are usually increased upon making the chain more polar in response to light exposure of photochrome side groups. We show here that a reverse response to light is obtained in azobenzene-modified amphiphilic polymers because of the fonnation of self-assemblies. We propose a model to calculate the cloud point accounting for the balance between inter- and intramolecular hydrophobic interactions. The principle illustrated in this study could help to tailor the polymer architecture for optimizing the magnitude of responses.