Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning

Michael Matthews, Mikayel Samvelyan, Jack Parker-holder, Edward Grefenstette, Tim Rocktäschel
Proceedings of The 1st Conference on Lifelong Learning Agents, PMLR 199:856-874, 2022.

Abstract

Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.

Cite this Paper

BibTeX
@InProceedings{pmlr-v199-matthews22a, title = {Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning}, author = {Matthews, Michael and Samvelyan, Mikayel and Parker-holder, Jack and Grefenstette, Edward and Rockt\"{a}schel, Tim}, booktitle = {Proceedings of The 1st Conference on Lifelong Learning Agents}, pages = {856--874}, year = {2022}, editor = {Chandar, Sarath and Pascanu, Razvan and Precup, Doina}, volume = {199}, series = {Proceedings of Machine Learning Research}, month = {22--24 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v199/matthews22a/matthews22a.pdf}, url = {https://proceedings.mlr.press/v199/matthews22a.html}, abstract = {Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.} }
Endnote
%0 Conference Paper %T Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning %A Michael Matthews %A Mikayel Samvelyan %A Jack Parker-holder %A Edward Grefenstette %A Tim Rocktäschel %B Proceedings of The 1st Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2022 %E Sarath Chandar %E Razvan Pascanu %E Doina Precup %F pmlr-v199-matthews22a %I PMLR %P 856--874 %U https://proceedings.mlr.press/v199/matthews22a.html %V 199 %X Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.
APA
Matthews, M., Samvelyan, M., Parker-holder, J., Grefenstette, E. & Rocktäschel, T.. (2022). Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning. Proceedings of The 1st Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 199:856-874 Available from https://proceedings.mlr.press/v199/matthews22a.html.

Related Material