The Logical Options Framework

Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan Decastro, Micah Fry, Daniela Rus
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:307-317, 2021.

Abstract

Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF’s learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps on our benchmarks. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-araki21a, title = {The Logical Options Framework}, author = {Araki, Brandon and Li, Xiao and Vodrahalli, Kiran and Decastro, Jonathan and Fry, Micah and Rus, Daniela}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {307--317}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/araki21a/araki21a.pdf}, url = {https://proceedings.mlr.press/v139/araki21a.html}, abstract = {Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF’s learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps on our benchmarks. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.} }
Endnote
%0 Conference Paper %T The Logical Options Framework %A Brandon Araki %A Xiao Li %A Kiran Vodrahalli %A Jonathan Decastro %A Micah Fry %A Daniela Rus %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-araki21a %I PMLR %P 307--317 %U https://proceedings.mlr.press/v139/araki21a.html %V 139 %X Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF’s learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps on our benchmarks. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.
APA
Araki, B., Li, X., Vodrahalli, K., Decastro, J., Fry, M. & Rus, D.. (2021). The Logical Options Framework. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:307-317 Available from https://proceedings.mlr.press/v139/araki21a.html.

Related Material