We propose a Markov-chain approximation method for discrete-time control problems, showing how to reap the speed gains from continuous-time algorithms in this class of models. Our approach specifies a discrete Markov chain on a grid, taking a first-order approximation of conditional...