We prepare monodisperse inverted (water-in-oil) emulsion droplets and observe their phase behavior in various types of nonaqueous continuous phases. By using linear polydimethylsiloxanes or linear alkanes of length n (monomer units), mixed at volume fraction phi with dodecane, we determine the boundary within the phi/n plane which describes the colloidal aggregation. In the limit of large n, the boundary is well predicted by a hard core depletion model of polymer chains. Deviations to this prediction become increasingly pronounced as n is reduced.