Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

Souradeep Dutta, Kaustubh Sridhar, Osbert Bastani, Edgar Dobriban, James Weimer, Insup Lee, Julia Parish-Morris
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1499-1509, 2023.

Abstract

Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms. A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider \emph{option templates}, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-dutta23a, title = {Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates}, author = {Dutta, Souradeep and Sridhar, Kaustubh and Bastani, Osbert and Dobriban, Edgar and Weimer, James and Lee, Insup and Parish-Morris, Julia}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1499--1509}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/dutta23a/dutta23a.pdf}, url = {https://proceedings.mlr.press/v205/dutta23a.html}, abstract = {Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms. A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider \emph{option templates}, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens} }
Endnote
%0 Conference Paper %T Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates %A Souradeep Dutta %A Kaustubh Sridhar %A Osbert Bastani %A Edgar Dobriban %A James Weimer %A Insup Lee %A Julia Parish-Morris %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-dutta23a %I PMLR %P 1499--1509 %U https://proceedings.mlr.press/v205/dutta23a.html %V 205 %X Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms. A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider \emph{option templates}, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens
APA
Dutta, S., Sridhar, K., Bastani, O., Dobriban, E., Weimer, J., Lee, I. & Parish-Morris, J.. (2023). Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1499-1509 Available from https://proceedings.mlr.press/v205/dutta23a.html.

Related Material