Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion

Roland Hafner, Tim Hertweck, Philipp Kloeppner, Michael Bloesch, Michael Neunert, Markus Wulfmeier, Saran Tunyasuvunakool, Nicolas Heess, Martin Riedmiller
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:1084-1099, 2021.

Abstract

Modern Reinforcement Learning (RL) algorithms promise to solve difficult motor control problems directly from raw sensory inputs. Their attraction is due in part to the fact that they can represent a general class of methods that allow to learn a solution with a reasonably set reward and minimal prior knowledge, even in situations where it is difficult or expensive for a human expert. For RL to truly make good on this promise, however, we need algorithms and learning setups that can work across a broad range of problems with minimal problem specific adjustments or engineering. In this paper, we study this idea of generality in the locomotion domain. We develop a learning framework that can learn sophisticated locomotion behaviour for a wide spectrum of legged robots, such as bipeds, tripeds, quadrupeds and hexapods, including wheeled variants. Our learning framework relies on a data-efficient, off-policy multi-task RL algorithm and a small set of reward functions that are semantically identical across robots. To underline the general applicability of the method, we keep the hyper-parameter settings and reward definitions constant across experiments and rely exclusively on on-board sensing. For nine different types of robots, including a real-world quadruped robot, we demonstrate that the same algorithm can rapidly learn diverse and reusable locomotion skills without any platform specific adjustments or additional instrumentation of the learning setup.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-hafner21a, title = {Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion}, author = {Hafner, Roland and Hertweck, Tim and Kloeppner, Philipp and Bloesch, Michael and Neunert, Michael and Wulfmeier, Markus and Tunyasuvunakool, Saran and Heess, Nicolas and Riedmiller, Martin}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {1084--1099}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/hafner21a/hafner21a.pdf}, url = {https://proceedings.mlr.press/v155/hafner21a.html}, abstract = {Modern Reinforcement Learning (RL) algorithms promise to solve difficult motor control problems directly from raw sensory inputs. Their attraction is due in part to the fact that they can represent a general class of methods that allow to learn a solution with a reasonably set reward and minimal prior knowledge, even in situations where it is difficult or expensive for a human expert. For RL to truly make good on this promise, however, we need algorithms and learning setups that can work across a broad range of problems with minimal problem specific adjustments or engineering. In this paper, we study this idea of generality in the locomotion domain. We develop a learning framework that can learn sophisticated locomotion behaviour for a wide spectrum of legged robots, such as bipeds, tripeds, quadrupeds and hexapods, including wheeled variants. Our learning framework relies on a data-efficient, off-policy multi-task RL algorithm and a small set of reward functions that are semantically identical across robots. To underline the general applicability of the method, we keep the hyper-parameter settings and reward definitions constant across experiments and rely exclusively on on-board sensing. For nine different types of robots, including a real-world quadruped robot, we demonstrate that the same algorithm can rapidly learn diverse and reusable locomotion skills without any platform specific adjustments or additional instrumentation of the learning setup.} }
Endnote
%0 Conference Paper %T Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion %A Roland Hafner %A Tim Hertweck %A Philipp Kloeppner %A Michael Bloesch %A Michael Neunert %A Markus Wulfmeier %A Saran Tunyasuvunakool %A Nicolas Heess %A Martin Riedmiller %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-hafner21a %I PMLR %P 1084--1099 %U https://proceedings.mlr.press/v155/hafner21a.html %V 155 %X Modern Reinforcement Learning (RL) algorithms promise to solve difficult motor control problems directly from raw sensory inputs. Their attraction is due in part to the fact that they can represent a general class of methods that allow to learn a solution with a reasonably set reward and minimal prior knowledge, even in situations where it is difficult or expensive for a human expert. For RL to truly make good on this promise, however, we need algorithms and learning setups that can work across a broad range of problems with minimal problem specific adjustments or engineering. In this paper, we study this idea of generality in the locomotion domain. We develop a learning framework that can learn sophisticated locomotion behaviour for a wide spectrum of legged robots, such as bipeds, tripeds, quadrupeds and hexapods, including wheeled variants. Our learning framework relies on a data-efficient, off-policy multi-task RL algorithm and a small set of reward functions that are semantically identical across robots. To underline the general applicability of the method, we keep the hyper-parameter settings and reward definitions constant across experiments and rely exclusively on on-board sensing. For nine different types of robots, including a real-world quadruped robot, we demonstrate that the same algorithm can rapidly learn diverse and reusable locomotion skills without any platform specific adjustments or additional instrumentation of the learning setup.
APA
Hafner, R., Hertweck, T., Kloeppner, P., Bloesch, M., Neunert, M., Wulfmeier, M., Tunyasuvunakool, S., Heess, N. & Riedmiller, M.. (2021). Towards General and Autonomous Learning of Core Skills: A Case Study in Locomotion. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:1084-1099 Available from https://proceedings.mlr.press/v155/hafner21a.html.

Related Material