Convergence of controlled models and finite-state approximation for discounted continuous-time Markov decision processes with constraints

In this paper we consider the convergence of a sequence {Mn} of the models of discounted continuous-time constrained Markov decision processes (MDP) to the “limit” one, denoted by M∞. For the models with denumerable states and unbounded transition rates, under reasonably mild conditions we prove that the (constrained) optimal policies and the optimal values of {Mn} converge to those of M∞, respectively, using a technique of occupation measures. As an application of the convergence result developed here, we show that an optimal policy and the optimal value for countable-state continuous-time MDP can be approximated by those of finite-state continuous-time MDP. Finally, we further illustrate such finite-state approximation by solving numerically a controlled birth-and-death system and also give the corresponding error bound of the approximation.