The determination of the content of impurities is a very frequent analysis performed on virgin olive oil samples, but the official method is quite work-intensive, and it would be convenient to have an alternative approximate method to evaluate the performance of the impurity removal process. In this work we develop a system based on computer vision and pattern recognition to classify the content of impurities of the olive oil samples in three sets, indicative of the goodness of the separation process of olive oil after its extraction from the paste. Starting from the histograms of the channels of the Red-Green-Blue (RGB), CIELAB and Hue-Saturation-Value (HSV) color spaces, we construct an initial input parameter vector and perform a feature extraction previous to the classification. Several linear and non-linear feature extraction techniques were evaluated, and the classifiers used were Support Vector Machines (SVMs) and Artificial Neural Networks (ANNs). The best classification rate achieved was 87.66%, obtained using Kernel Principal Components Analysis (KPCA) and a grade-3-polynomial kernel SVM. The best result using ANNs was 82.38%, yielded by the use of Principal Component Analysis (PCA) with the Perceptron. (C) 2013 Elsevier Ltd. All rights reserved.