Regret-based Reward Elicitation for Markov Decision Processes

Kevin Regan, Craig Boutilier
Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, PMLR R7:452-459, 2009.

Abstract

The specification of aMarkov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR7-regan09a, title = {Regret-based Reward Elicitation for {M}arkov Decision Processes}, author = {Regan, Kevin and Boutilier, Craig}, booktitle = {Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence}, pages = {452--459}, year = {2009}, editor = {Bilmes, Jeff and Ng, Andrew Y.}, volume = {R7}, series = {Proceedings of Machine Learning Research}, month = {18--21 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r7/main/assets/regan09a/regan09a.pdf}, url = {https://proceedings.mlr.press/r7/regan09a.html}, abstract = {The specification of aMarkov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Regret-based Reward Elicitation for Markov Decision Processes %A Kevin Regan %A Craig Boutilier %B Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2009 %E Jeff Bilmes %E Andrew Y. Ng %F pmlr-vR7-regan09a %I PMLR %P 452--459 %U https://proceedings.mlr.press/r7/regan09a.html %V R7 %X The specification of aMarkov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function. %Z Reissued by PMLR on 04 October 2026.
APA
Regan, K. & Boutilier, C.. (2009). Regret-based Reward Elicitation for Markov Decision Processes. Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R7:452-459 Available from https://proceedings.mlr.press/r7/regan09a.html. Reissued by PMLR on 04 October 2026.

Related Material