Physical fault detection in panels that are part of photovoltaic (PV) plants typically involves the analysis of thermal and electroluminescent images, which makes it either difficult or impossible to identify the source of the fault in the plant. This paper proposes a method of automatic physical fault classification for PV plants using convolutional neural networks for semantic segmentation and classification from RGB images. This study shows experimental results for 2 output classes identified as a fault and no fault, and 4 output classes as no fault, cracks, shadows, and dust that cannot be easily detected. The proposed method presents an average accuracy of 75% for 2 output classes and 70% for 4 classes, showing a positive approach to the proposed classification method for PV systems. (c) 2020 Elsevier Ltd. All rights reserved.