We investigated the microscopic structure of the nonswellable hydrogel using small-angle neutron scattering (SANS). The hydrogel consisted of four-armed thermoresponsive prepolymer units embedded in a homogeneous network of four-armed poly(ethylene glycol) (Tetra-PEG). The structure of the hydrogel was similar to that of the ordinary Tetra-PEG hydrogels at temperatures below 16.6 degrees C, whereas discrete spherical domains were formed at temperatures above 19.5 degrees C. The number of prepolymer units contained in one domain was much larger than unity, indicating that multiple thermoresponsive prepolymer units as well as Tetra-PEG units gathered to form a domain. Formation of domains much larger than a single prepolymer unit led to significant frustration of the matrix polymer network outside the domains. This This frustration enhanced the elastic energy of the matrix network which would cancel the osmotic pressure and induce significant macroscopic shrinking. The selection mechanism of the domain size could qualitatively be explained by the balance between the interfacial and conformational free energies.