We consider a minimizing risk model in a stochastic shortest path problem in which for each node of a graph we select a probability distribution over the set of successor nodes so as to reach a given target node with minimum threshold probability. We formulate such a problem as undiscounted...

The aim of this paper is to investigate the Lagrangian approach and a related Linear Programming (LP) that appear in constrained Markov decision processes (CMDPs) with a countable state space and total expected cost criteria (of which the expected discounted cost is a special case). We consider...

We develop an algorithm to compute optimal policies for Markov decision processes subject to constraints that result from some observability restrictions on the process. We assume that the state of the Markov process is unobservable. There is an observable process related to the unobservable...

We consider eight problems in which we maximize or minimize threshold probabilities in discounted Markov decision processes with bounded reward set. We show that such problems are classified to two equivalence classes and give a relationship between optimal values and optimal policies of...

For a vector-valued Markov decision process with discounted reward criterion, we study the structure of its value spaces defined for all initial states. At first we discuss the relationship between the value spaces, i.e. we verify a linking property for optimality. We next show that if the...

We introduce the time-consistency concept that is inspired by the so-called “principle of optimality” of dynamic programming and demonstrate – via an example – that the conditional value-at-risk (CVaR) need not be time-consistent in a multi-stage case. Then, we give the formulation of...

Impulsive control consideres so called interventions meaning immediate change of the state of the system; between intervention, the continuous time jump Markov process is uncontrollable, with “natural” jump intensities. Multicriteria control problem for such model is considered, and the...

The aim of the paper is to show that Lyapunov-like ergodicity conditions on Markov decision processes with Borel state space and possibly unbounded cost provide the approximation of an average cost optimal policy by solvingn-stage optimization problems (n=1, 2, ...). The used approach ensures...

The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term ’risk-sensitive’ refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure...