|
|
Central Moments and Risk-Sensitive Optimality in Markov Reward Processes
Sladký, Karel
In this note we consider discrete- and continuous-time Markov decision processes with finite state space. There is no doubt that usual optimality criteria examined in the literature on optimization of Markov reward processes, e.g. total discounted or mean reward, may be quite insufficient to select more sophisticated criteria that reflect also the variability-risk features of the problem. In this note we focus on models where the stream of rewards generated by the Markov process is evaluated by an exponential utility function with a given risk sensitivity coefficient (so-called risk-sensitive models).For the risk sensitive case, i.e. if the considered risk-sensitivity coefficient is non-zero, we establish explicit formulas for growth rate of expectation of the exponential utility function. Recall that in this case along with the total reward also it higher moments are taken into account. Using Taylor expansion of the utility function we present explicit formulas for calculating variance a higher central moments of the total reward generated by the |Markov reward process along with its asymptotic behaviour.
|
|
|
Average Reward Optimality in Semi-Markov Decision Processes with Costly Interventions
Sladký, Karel
In this note we consider semi-Markov reward decision processes evolving on finite state spaces. We focus attention on average reward models, i.e. we establish explicit formulas for the growth rate of the total expected reward. In contrast to the standard models we assume that the decision maker can also change the running process by some (costly) intervention. Recall that the result for optimality criteria for the classical Markov decision chains in discrete and continuous time setting turn out to be a very specific case of the considered model. The aim is to formulate optimality conditions for semi-Markov models with interventions and present algorithmical procedures for finding optimal solutions.
|
|
|
Apartment building in Vracov town
Sladký, Karel ; Žajdlík, Tomáš (referee) ; Šuhajdová, Eva (advisor)
The subject of the bachelor thesis is the elaboration of project documentation for a new residential building in Vracov. The building is designed as a three-storey fully basemented building. In the underground floor there is a common room, a whirlpool and a cellar. On the above-ground floors there are apartments for permanent living. 6 residential units have been designed. The above-ground apartments have loggias. The building has a flat roof with a duck. The structural system of the building is mainly made of wood cement fittings filled with static concrete. The building is insulated with ETICS system above the first floor. The building has an elevator shaft. Horizontal load-bearing structures are designed reinforced concrete floor slabs. The whole building is based on foundation strips. The facade is made of white paint and imitation wood. The bachelor thesis contains the project documentation for the implementation of the construction.
|
|
|
Higher - order Markov chains and applications in econometrics
Straňáková, Alena ; Prášková, Zuzana (advisor) ; Sladký, Karel (referee)
In this paper, we generalize Raftery's model of Markov chain to a higher-order multivariate Markov chain model. This model is more suitable for practical applications because of smaller number of independent parameters. We propose a method of estimation of parameters of the model and apply it to the Credit risk measuring of a portfolio. We compute Value at Risk and Expected Shortfall in this portfolio. Theoretical results are applied to real data.
|
|
| |
|
|
Stochastic Dynamic Programming Problems: Theory and Applications.
Lendel, Gabriel ; Sladký, Karel (advisor) ; Lachout, Petr (referee)
Title: Stochastic Dynamic Programming Problems: Theory and Applications Author: Gabriel Lendel Department: Department of Probability and Mathematical Statistics Supervisor: Ing. Karel Sladký CSc. Supervisor's e-mail address: sladky@utia.cas.cz Abstract: In the present work we study Markov decision processes which provide a mathematical framework for modeling decision-making in situations where outcomes are partly random and partly under the control of a decision maker. We study iterative procedures for finding policy that is optimal or nearly optimal with respect to the selec- ted criteria. Specifically, we mainly examine the task of finding a policy that is optimal with respect to the total expected discounted reward or the average expected reward for discrete or continuous systems. In the work we study policy iteration algorithms and aproximative value iteration algorithms. We give numerical analysis of specific problems. Keywords: Stochastic dynamic programming, Markov decision process, policy ite- ration, value iteration
|
|
|
Risk-Sensitivity and Average Optimality in Markov and Semi-Markov Reward Processes
Sladký, Karel
This contribution is devoted to risk-sensitivity in long-run average optimality of Markov and semi-Markov reward processes. Since the traditional average optimality criteria cannot reflect the variability-risk features of the problem, we are interested in more sophisticated approaches where the stream of rewards generated by the Markov chain that is evaluated by an exponential utility function with a given risk sensitivity coefficient. Recall that for the risk sensitivity coefficient equal to zero (i.e. the so called risk-neutral case) we arrive at traditional optimality criteria, if the risk sensitivity coefficient is close to zero the Taylor expansion enables to evaluate variability of the generated total reward. Observe that the first moment of the total reward corresponds to expectation of total reward and the second central moment to the reward variance. In this note we present necessary and sufficient risk-sensitivity and risk-neutral optimality conditions for long run risk-sensitive average optimality criterion of unichain Markov and semi-Markov reward processes.
|
|
|
Second Order Optimality in Markov and Semi-Markov Decision Processes
Sladký, Karel
Semi-Markov decision processes can be considered as an extension of discrete- and continuous-time Markov reward models. Unfortunately, traditional optimality criteria as long-run average reward per time may be quite insufficient to characterize the problem from the point of a decision maker. To this end it may be preferable if not necessary to select more sophisticated criteria that also reflect variability-risk features of the problem. Perhaps the best known approaches stem from the classical work of Markowitz on mean-variance selection rules, i.e. we optimize the weighted sum of average or total reward and its variance. Such approach has been already studied for very special classes of semi-Markov decision processes, in particular, for Markov decision processes in discrete - and continuous-time setting. In this note these approaches are summarized and possible extensions to the wider class of semi-Markov decision processes is discussed. Attention is mostly restricted to uncontrolled models in which the chain is aperiodic and contains a single class of recurrent states. Considering finite time horizons, explicit formulas for the first and second moments of total reward as well as for the corresponding variance are produced.
|
|
|
Risk-sensitive and Mean Variance Optimality in Continuous-time Markov Decision Chains
Sladký, Karel
In this note we consider continuous-time Markov decision processes with finite state and actions spaces where the stream of rewards generated by the Markov processes is evaluated by an exponential utility function with a given risk sensitivitycoefficient (so-called risk-sensitive models). If the risk sensitivity coefficient equals zero (risk-neutral case) we arrive at a standard Markov decision process. Then we can easily obtain necessary and sufficient mean reward optimality conditions and the variability can be evaluated by the mean variance of total expected rewards. For the risk-sensitive case, i.e. if the risk-sensitivity coefficient is non-zero, for a given value of the risk-sensitivity coefficient we establish necessary and sufficient optimality conditions for maximal (or minimal) growth rate of expectation of the exponential utility function, along with mean value of the corresponding certainty equivalent. Recall that in this case along with the total reward also its higher moments are taken into account.
|
|
|
Risk-Sensitive Optimality in Markov Games
Sladký, Karel ; Martínez Cortés, V. M.
The article is devoted to risk-sensitive optimality in Markov games. Attention is focused on Markov games evolving on communicating Markov chains with two-players with opposite aims. Considering risk-sensitive optimality criteria means that total reward generated by the game is evaluated by exponential utility function with a given risk-sensitive coefficient. In particular, the first player (resp. the secondplayer) tries to maximize (resp. minimize) the long-run risk sensitive average reward. Observe that if the second player is dummy, the problem is reduced to finding optimal policy of the Markov decision chain with the risk-sensitive optimality. Recall that for the risk sensitivity coefficient equal to zero we arrive at traditional optimality criteria. In this article, connections between risk-sensitive and risk-neutral Markov decisionchains and Markov games models are studied using discrepancy functions. Explicit formulae for bounds on the risk-sensitive average long-run reward are reported. Policy iteration algorithm for finding suboptimal policies of both players is suggested. The obtained results are illustrated on numerical example.
|
guest ::
