The heterogeneity and micropores distribution of coal and shale reservoirs make them difficult to be quantitatively characterized by quick and economical approaches. A new method based on the atomic force microscopy (AFM) test is applied to reveal the pore structure and surface roughness of coal and shale with the help of open source Gwyddion software. The surface morphologies of coal and shale are reconstructed in two and three dimensions by AFM data, and the surface roughness is evaluated by parameters like mean roughness, mean square roughness, surface skewness, and kurtosis coefficient. Pores of different scales are quantitatively recognized by the Threshold and Watershed methods, and the two methods are combined for more accurate results. The Threshold method is appropriate for characterizing pores of 10-500 nm and the Watershed method is more suitable for 1-200 nm pores. Parameters such as pore numbers, plane porosity, pore size distribution, pore surface area and pore volume are obtained based on the combined method in revealing pores of 1-500 nm range. The tested samples are also evaluated by scanning electron microscope images, which shows a reliable consistency between the pore scales compared with the AFM results. The method, using easily accessed AFM experiment and open source software, can be widely used in characterizing the surface roughness and pore structure of unconventional oil and gas reservoirs.