The exponential growth of solar energy poses challenges to power systems, mostly due to its uncertain and variable characteristics. Hence, solar forecasting, such as very short-term solar forecasting (VSTSF), has been widely adopted to assist power system operations. The VSTSF takes inputs from various sources, among which sky image-based VSTSF is not yet well-studied compared to its counterparts. In this paper, a deep convolutional neural network (CNN) model, called the SolarNet, is developed to forecast the operational 1-h-ahead global horizontal irradiance (GHI) by only using sky images without numerical measurements and extra feature engineering. The SolarNet has a set of models that generate fixed-step GHI in parallel. Each model is composed of 20 convolutional, max-pooling, and fully-connected layers, which learns latent patterns between sky images and GHI in an end-to-end manner. Numerical results based on six years data show that the developed SolarNet outperforms the benchmarking persistence of cloudiness model and machine learning models with an 8.85% normalized root mean square error and a 25.14% forecasting skill score. The SolarNet shows superiority under various weather conditions.