Risk-Aware Active Inverse Reinforcement Learning

Daniel S. Brown, Yuchen Cui, Scott Niekum
Proceedings of The 2nd Conference on Robot Learning, PMLR 87:362-372, 2018.

Abstract

Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v87-brown18a, title = {Risk-Aware Active Inverse Reinforcement Learning}, author = {Brown, Daniel S. and Cui, Yuchen and Niekum, Scott}, booktitle = {Proceedings of The 2nd Conference on Robot Learning}, pages = {362--372}, year = {2018}, editor = {Billard, Aude and Dragan, Anca and Peters, Jan and Morimoto, Jun}, volume = {87}, series = {Proceedings of Machine Learning Research}, month = {29--31 Oct}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v87/brown18a/brown18a.pdf}, url = {https://proceedings.mlr.press/v87/brown18a.html}, abstract = {Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task. } }
Endnote
%0 Conference Paper %T Risk-Aware Active Inverse Reinforcement Learning %A Daniel S. Brown %A Yuchen Cui %A Scott Niekum %B Proceedings of The 2nd Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2018 %E Aude Billard %E Anca Dragan %E Jan Peters %E Jun Morimoto %F pmlr-v87-brown18a %I PMLR %P 362--372 %U https://proceedings.mlr.press/v87/brown18a.html %V 87 %X Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task.
APA
Brown, D.S., Cui, Y. & Niekum, S.. (2018). Risk-Aware Active Inverse Reinforcement Learning. Proceedings of The 2nd Conference on Robot Learning, in Proceedings of Machine Learning Research 87:362-372 Available from https://proceedings.mlr.press/v87/brown18a.html.

Related Material