Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics

Krishan Rana, Ming Xu, Brendan Tidd, Michael Milford, Niko Suenderhauf
Proceedings of The 6th Conference on Robot Learning, PMLR 205:2095-2104, 2023.

Abstract

Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this \textit{skill space} is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only \textit{sampling} skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained \textit{skill adaptation} enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accelerating exploration, outperforming prior works.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-rana23a, title = {Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics}, author = {Rana, Krishan and Xu, Ming and Tidd, Brendan and Milford, Michael and Suenderhauf, Niko}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {2095--2104}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/rana23a/rana23a.pdf}, url = {https://proceedings.mlr.press/v205/rana23a.html}, abstract = {Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this \textit{skill space} is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only \textit{sampling} skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained \textit{skill adaptation} enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accelerating exploration, outperforming prior works.} }
Endnote
%0 Conference Paper %T Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics %A Krishan Rana %A Ming Xu %A Brendan Tidd %A Michael Milford %A Niko Suenderhauf %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-rana23a %I PMLR %P 2095--2104 %U https://proceedings.mlr.press/v205/rana23a.html %V 205 %X Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this \textit{skill space} is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only \textit{sampling} skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained \textit{skill adaptation} enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accelerating exploration, outperforming prior works.
APA
Rana, K., Xu, M., Tidd, B., Milford, M. & Suenderhauf, N.. (2023). Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:2095-2104 Available from https://proceedings.mlr.press/v205/rana23a.html.

Related Material