Environment Design for Inverse Reinforcement Learning

Thomas Kleine Buening, Victor Villin, Christos Dimitrakakis
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:24808-24828, 2024.

Abstract

Learning a reward function from demonstrations suffers from low sample-efficiency. Even with abundant data, current inverse reinforcement learning methods that focus on learning from a single environment can fail to handle slight changes in the environment dynamics. We tackle these challenges through adaptive environment design. In our framework, the learner repeatedly interacts with the expert, with the former selecting environments to identify the reward function as quickly as possible from the expert’s demonstrations in said environments. This results in improvements in both sample-efficiency and robustness, as we show experimentally, for both exact and approximate inference.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-kleine-buening24a, title = {Environment Design for Inverse Reinforcement Learning}, author = {Kleine Buening, Thomas and Villin, Victor and Dimitrakakis, Christos}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {24808--24828}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/kleine-buening24a/kleine-buening24a.pdf}, url = {https://proceedings.mlr.press/v235/kleine-buening24a.html}, abstract = {Learning a reward function from demonstrations suffers from low sample-efficiency. Even with abundant data, current inverse reinforcement learning methods that focus on learning from a single environment can fail to handle slight changes in the environment dynamics. We tackle these challenges through adaptive environment design. In our framework, the learner repeatedly interacts with the expert, with the former selecting environments to identify the reward function as quickly as possible from the expert’s demonstrations in said environments. This results in improvements in both sample-efficiency and robustness, as we show experimentally, for both exact and approximate inference.} }
Endnote
%0 Conference Paper %T Environment Design for Inverse Reinforcement Learning %A Thomas Kleine Buening %A Victor Villin %A Christos Dimitrakakis %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-kleine-buening24a %I PMLR %P 24808--24828 %U https://proceedings.mlr.press/v235/kleine-buening24a.html %V 235 %X Learning a reward function from demonstrations suffers from low sample-efficiency. Even with abundant data, current inverse reinforcement learning methods that focus on learning from a single environment can fail to handle slight changes in the environment dynamics. We tackle these challenges through adaptive environment design. In our framework, the learner repeatedly interacts with the expert, with the former selecting environments to identify the reward function as quickly as possible from the expert’s demonstrations in said environments. This results in improvements in both sample-efficiency and robustness, as we show experimentally, for both exact and approximate inference.
APA
Kleine Buening, T., Villin, V. & Dimitrakakis, C.. (2024). Environment Design for Inverse Reinforcement Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:24808-24828 Available from https://proceedings.mlr.press/v235/kleine-buening24a.html.

Related Material