To solve a time-varying overdetermined problem, a novel varying-parameter convergent-differential neural network (VP-CDNN) is proposed, designed, and discussed. Specifically, a vector-error function is first defined. Second, according to neural dynamic design method, an implicit-dynamic equation with a time-varying parameter is derived. Mathematics analysis and theoretical proof verify that the VP-CDNN can obtain the least-squares solution with a super exponential convergence rate. In addition, it is also proved that VP-CDNN can restrain the noise efficiently. Simulations among the VP-CDNN, gradient-based recurrent neural network and zeroing neural network verify that the VP-CDNN has faster speed, higher accuracy, and stronger robustness. At last, applications to data fitting and system identification further verify the high effectiveness and efficiency of the VP-CDNN.