An adaptive control algorithm is presented for constrained finite controlled Markov chains with unknown transition probabilities. A finite set of algebraic constraints has been considered. The Lagrange multipliers approach is used to solve this constrained optimization problem. This scheme is such that at each time n estimates the control policy on the basis on Bush-Mosteller scheme which is related to stochastic approximation procedures. We present the asymptotic properties (convergence and order of convergence rate) of the algorithm. They follow from the law of dependent large numbers, martingales theory and Lyapunov function analysis approaches.