Solution Methods for Constrained Markov Decision Process with Continuous Probability Modulation

Marek Petrik, Dharmashankar Subramanian, Janusz Marecki
Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, PMLR R11:599-607, 2013.

Abstract

We propose solution methods for previously- unsolved constrained MDPs in which actions can continuously modify the transition probabilities within some acceptable sets. While many meth- ods have been proposed to solve regular MDPs with large state sets, there are few practical approaches for solving constrained MDPs with large action sets. In particular, we show that the continuous action sets can be replaced by their extreme points when the rewards are linear in the modulation. We also develop a tractable opti- mization formulation for concave reward func- tions and, surprisingly, also extend it to non- concave reward functions by using their concave envelopes. We evaluate the effectiveness of the approach on the problem of managing delinquen- cies in a portfolio of loans.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR11-petrik13a, title = {Solution Methods for Constrained {M}arkov Decision Process with Continuous Probability Modulation}, author = {Petrik, Marek and Subramanian, Dharmashankar and Marecki, Janusz}, booktitle = {Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence}, pages = {599--607}, year = {2013}, editor = {Nicholson, Ann and Smyth, Padhraic}, volume = {R11}, series = {Proceedings of Machine Learning Research}, month = {12--14 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r11/main/assets/petrik13a/petrik13a.pdf}, url = {https://proceedings.mlr.press/r11/petrik13a.html}, abstract = {We propose solution methods for previously- unsolved constrained MDPs in which actions can continuously modify the transition probabilities within some acceptable sets. While many meth- ods have been proposed to solve regular MDPs with large state sets, there are few practical approaches for solving constrained MDPs with large action sets. In particular, we show that the continuous action sets can be replaced by their extreme points when the rewards are linear in the modulation. We also develop a tractable opti- mization formulation for concave reward func- tions and, surprisingly, also extend it to non- concave reward functions by using their concave envelopes. We evaluate the effectiveness of the approach on the problem of managing delinquen- cies in a portfolio of loans.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Solution Methods for Constrained Markov Decision Process with Continuous Probability Modulation %A Marek Petrik %A Dharmashankar Subramanian %A Janusz Marecki %B Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2013 %E Ann Nicholson %E Padhraic Smyth %F pmlr-vR11-petrik13a %I PMLR %P 599--607 %U https://proceedings.mlr.press/r11/petrik13a.html %V R11 %X We propose solution methods for previously- unsolved constrained MDPs in which actions can continuously modify the transition probabilities within some acceptable sets. While many meth- ods have been proposed to solve regular MDPs with large state sets, there are few practical approaches for solving constrained MDPs with large action sets. In particular, we show that the continuous action sets can be replaced by their extreme points when the rewards are linear in the modulation. We also develop a tractable opti- mization formulation for concave reward func- tions and, surprisingly, also extend it to non- concave reward functions by using their concave envelopes. We evaluate the effectiveness of the approach on the problem of managing delinquen- cies in a portfolio of loans. %Z Reissued by PMLR on 04 October 2026.
APA
Petrik, M., Subramanian, D. & Marecki, J.. (2013). Solution Methods for Constrained Markov Decision Process with Continuous Probability Modulation. Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R11:599-607 Available from https://proceedings.mlr.press/r11/petrik13a.html. Reissued by PMLR on 04 October 2026.

Related Material