Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

Andreas Schlaginhaufen, Maryam Kamgarpour
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:30224-30251, 2023.

Abstract

Two main challenges in Reinforcement Learning (RL) are designing appropriate reward functions and ensuring the safety of the learned policy. To address these challenges, we present a theoretical framework for Inverse Reinforcement Learning (IRL) in constrained Markov decision processes. From a convex-analytic perspective, we extend prior results on reward identifiability and generalizability to both the constrained setting and a more general class of regularizations. In particular, we show that identifiability up to potential shaping (Cao et al., 2021) is a consequence of entropy regularization and may generally no longer hold for other regularizations or in the presence of safety constraints. We also show that to ensure generalizability to new transition laws and constraints, the true reward must be identified up to a constant. Additionally, we derive a finite sample guarantee for the suboptimality of the learned rewards, and validate our results in a gridworld environment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-schlaginhaufen23a, title = {Identifiability and Generalizability in Constrained Inverse Reinforcement Learning}, author = {Schlaginhaufen, Andreas and Kamgarpour, Maryam}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {30224--30251}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/schlaginhaufen23a/schlaginhaufen23a.pdf}, url = {https://proceedings.mlr.press/v202/schlaginhaufen23a.html}, abstract = {Two main challenges in Reinforcement Learning (RL) are designing appropriate reward functions and ensuring the safety of the learned policy. To address these challenges, we present a theoretical framework for Inverse Reinforcement Learning (IRL) in constrained Markov decision processes. From a convex-analytic perspective, we extend prior results on reward identifiability and generalizability to both the constrained setting and a more general class of regularizations. In particular, we show that identifiability up to potential shaping (Cao et al., 2021) is a consequence of entropy regularization and may generally no longer hold for other regularizations or in the presence of safety constraints. We also show that to ensure generalizability to new transition laws and constraints, the true reward must be identified up to a constant. Additionally, we derive a finite sample guarantee for the suboptimality of the learned rewards, and validate our results in a gridworld environment.} }
Endnote
%0 Conference Paper %T Identifiability and Generalizability in Constrained Inverse Reinforcement Learning %A Andreas Schlaginhaufen %A Maryam Kamgarpour %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-schlaginhaufen23a %I PMLR %P 30224--30251 %U https://proceedings.mlr.press/v202/schlaginhaufen23a.html %V 202 %X Two main challenges in Reinforcement Learning (RL) are designing appropriate reward functions and ensuring the safety of the learned policy. To address these challenges, we present a theoretical framework for Inverse Reinforcement Learning (IRL) in constrained Markov decision processes. From a convex-analytic perspective, we extend prior results on reward identifiability and generalizability to both the constrained setting and a more general class of regularizations. In particular, we show that identifiability up to potential shaping (Cao et al., 2021) is a consequence of entropy regularization and may generally no longer hold for other regularizations or in the presence of safety constraints. We also show that to ensure generalizability to new transition laws and constraints, the true reward must be identified up to a constant. Additionally, we derive a finite sample guarantee for the suboptimality of the learned rewards, and validate our results in a gridworld environment.
APA
Schlaginhaufen, A. & Kamgarpour, M.. (2023). Identifiability and Generalizability in Constrained Inverse Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:30224-30251 Available from https://proceedings.mlr.press/v202/schlaginhaufen23a.html.

Related Material