An iterative procedure for constructing subsolutions of deterministic or stochastic optimal control problems in discrete time with continuous state space is introduced. The procedure generates a nondecreasing sequence of subsolutions, giving true lower bounds on the minimal costs. Convergence of the values at any fixed initial state is shown.