Mixtures of alkali swellable microgels and linear PNIPAm chains exhibit doubly responsive properties both with pH and temperature. Below the lower critical solution temperature (LCST), the linear chains of PNIPAm are soluble and increase the osmotic pressure outside the microgels, which causes them to deswell. Above the LCST, the PNIPAm chains become insoluble and form spherical colloidal particles confined between the microgels that subsequently reswell. The swelling and deswelling of the microgels change the theological proper-ties of the composites, providing a unique way to tune the elasticity of the composites with temperature. The structure of the composites above the LCST is studied using multiple light scattering and fluorescence confocal microscopy. The phase separation of PNIPAm above the LCST is strongly affected by the confinement of the PNIPAm chains between the microgels.