Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning

Jan Achterhold, Markus Krimmel, Joerg Stueckler
Proceedings of The 6th Conference on Robot Learning, PMLR 205:225-236, 2023.

Abstract

Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment’s state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-achterhold23a, title = {Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning}, author = {Achterhold, Jan and Krimmel, Markus and Stueckler, Joerg}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {225--236}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/achterhold23a/achterhold23a.pdf}, url = {https://proceedings.mlr.press/v205/achterhold23a.html}, abstract = {Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment’s state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.} }
Endnote
%0 Conference Paper %T Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning %A Jan Achterhold %A Markus Krimmel %A Joerg Stueckler %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-achterhold23a %I PMLR %P 225--236 %U https://proceedings.mlr.press/v205/achterhold23a.html %V 205 %X Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment’s state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.
APA
Achterhold, J., Krimmel, M. & Stueckler, J.. (2023). Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:225-236 Available from https://proceedings.mlr.press/v205/achterhold23a.html.

Related Material