Partially quaternized poly(4-vinylpyridine) (P4VP) gels were prepared by irradiating aqueous solutions with gamma-rays. Swelling degrees in water increased with increasing the degree of quaternization (charge density) except for gels of I- or SCN- counterions, while those in mixed solvents (water/methanol, ethanol, 2-propanol, dimethyl sulfoxide, acetone, or dioxane) decreased with increasing the charge density and/or with decreasing the dielectric constant of the mixed solvent. This trend was more evident for counterion species having stronger affinity to quaternary ammonium group, e.g., I- and SCN-, for which very sharp deswelling, comparable to volume phase transition, was observed even in water. A remarkable finding was that the sharp collapse occurred at almost a constant dielecric constant (D) irrespective of the solvent species; e.g., when the counterion was Cl-, the collapse was observed at D = 58 +/- 2 for 50% quaternized gel and at 71 or 72 for the fully quaternized one. Effects of cross-linking, counterion binding, polymer charge density, and dielectric constant of solvent on the gel collapse are discussed.