The effect of concentration of Er3+ on the up-conversion and photoluminescence properties of Gd1.00-xErxNbO4, x=0-0.50 which has monoclinic fergusonite-type structure as a main phase has been investigated, using a processing technique based on hydrothermal method. Under weakly basic hydrothermal condition at 240 degrees C for 5hours, a single phase of fergusonite-type Gd1.00-xErxNbO4 solid solution was directly formed as nanocrystals by the substitutional incorporation of Er3+ into GdNbO4 because of the gradual and linear decrease in the lattice parameters of the monoclinic phase corresponding to the Vegard's Law. The gadolinium niobate doped with 2mol% Er3+, Gd0.98Er0.02NbO4 after heating at 1300 degrees C for 1hour, which has nanocrystalline structure whose crystallite size is around 29nm, exhibits the highest photoluminescence intensity in the green spectral region, 515-560nm under excitation at wavelength of 254nm. On the other hand, the up-converted luminescence intensity of the niobate nanocrystals becomes the maximum at the concentration of 20mol% Er3+, Gd0.80Er0.20NbO4 under excitation at 980nm. These results demonstrate that the material, Er3+-doped GdNbO4 nanocrystals prepared through hydrothermal route and postheating has potential for up-converting phosphor.