Two color-memory experiments were performed to investigate whether observers tended to confuse colors with a smaller color difference in memory or colors in a same color-category region. We made color stimuli on a color CRT. Color difference was determined by a simultaneous color discrimination experiment. Color-category regions were obtained by a categorical color-naming experiment using the 11 basic color names: white, black, red, green, yellow, blue, brown, orange, purple, pink, and gray. The results show that two colors with a certain color difference can be confused more easily when they are in a same color category than in different color categories, and that colors identified with memory tend to distribute within their own color-category regions or their neighbor color-category regions, depending on their positions in a color space. These findings indicate that color memory is characterized by the color categories, suggesting a color-category mechanism in a higher level of color vision.