This article proposed a novel approach to color measurement of a single yarn using hyperspectral imaging system (HIS). Due to the size of a single yarn, it is impossible for spectrophotometers to measure its color directly. The HIS can acquire the spectral reflectance of continuous bands within a region of interest on a yarn sample, which can achieve color measurement of a single yarn compared with traditional spectrophotometers. A single yarn is segmented from the background by a spectral matching method through adaptively setting threshold of Frechet distance values. The spectral reflectance of single yarn is specified by a method that lightness of pixels used as weight. The experiment based on Pantone Cotton Chip Set shows that the interinstrument agreement between the HIS and a standard spectrophotometer Datacolor SF650 has a significant improvement after using the R-Model, and the average percentage improvement of the color difference is up to 54.99%. The yarn segmentation comparative experimental results show that the proposed method to segment single yarn from background is better in retaining the edge information of the yarn than the modified K-means clustering method, and the color of the yarn segmented by the proposed method is more similar to the actual color of single yarn.