Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion

Michael Bloesch, Jan Humplik, Viorica Patraucean, Roland Hafner, Tuomas Haarnoja, Arunkumar Byravan, Noah Yamamoto Siegel, Saran Tunyasuvunakool, Federico Casarini, Nathan Batchelor, Francesco Romano, Stefano Saliceti, Martin Riedmiller, S. M. Ali Eslami, Nicolas Heess
Proceedings of the 5th Conference on Robot Learning, PMLR 164:1502-1511, 2022.

Abstract

Algorithms for self-learning systems have made considerable progress in recent years, yet safety concerns and the need for additional instrumentation have so far largely limited learning experiments with real robots to well controlled lab settings. In this paper, we demonstrate how a small bipedal robot can autonomously learn to walk with minimal human intervention and with minimal instrumentation of the environment. We employ data-efficient off-policy deep reinforcement learning to learn to walk end-to-end, directly on hardware, using rewards that are computed exclusively from proprioceptive sensing. To allow the robot to autonomously adapt its behaviour to its environment, we additionally provide the agent with raw RGB camera images as input. By deploying two robots in different geographic locations while sharing data in a distributed learning setup, we achieve higher throughput and greater diversity of the training data. Our learning experiments constitute a step towards the long-term vision of learning “in the wild” for legged robots, and, to our knowledge, represent the first demonstration of learning a deep neural network controller for bipedal locomotion directly on hardware.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-bloesch22a, title = {Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion}, author = {Bloesch, Michael and Humplik, Jan and Patraucean, Viorica and Hafner, Roland and Haarnoja, Tuomas and Byravan, Arunkumar and Siegel, Noah Yamamoto and Tunyasuvunakool, Saran and Casarini, Federico and Batchelor, Nathan and Romano, Francesco and Saliceti, Stefano and Riedmiller, Martin and Eslami, S. M. Ali and Heess, Nicolas}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {1502--1511}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/bloesch22a/bloesch22a.pdf}, url = {https://proceedings.mlr.press/v164/bloesch22a.html}, abstract = {Algorithms for self-learning systems have made considerable progress in recent years, yet safety concerns and the need for additional instrumentation have so far largely limited learning experiments with real robots to well controlled lab settings. In this paper, we demonstrate how a small bipedal robot can autonomously learn to walk with minimal human intervention and with minimal instrumentation of the environment. We employ data-efficient off-policy deep reinforcement learning to learn to walk end-to-end, directly on hardware, using rewards that are computed exclusively from proprioceptive sensing. To allow the robot to autonomously adapt its behaviour to its environment, we additionally provide the agent with raw RGB camera images as input. By deploying two robots in different geographic locations while sharing data in a distributed learning setup, we achieve higher throughput and greater diversity of the training data. Our learning experiments constitute a step towards the long-term vision of learning “in the wild” for legged robots, and, to our knowledge, represent the first demonstration of learning a deep neural network controller for bipedal locomotion directly on hardware.} }
Endnote
%0 Conference Paper %T Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion %A Michael Bloesch %A Jan Humplik %A Viorica Patraucean %A Roland Hafner %A Tuomas Haarnoja %A Arunkumar Byravan %A Noah Yamamoto Siegel %A Saran Tunyasuvunakool %A Federico Casarini %A Nathan Batchelor %A Francesco Romano %A Stefano Saliceti %A Martin Riedmiller %A S. M. Ali Eslami %A Nicolas Heess %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-bloesch22a %I PMLR %P 1502--1511 %U https://proceedings.mlr.press/v164/bloesch22a.html %V 164 %X Algorithms for self-learning systems have made considerable progress in recent years, yet safety concerns and the need for additional instrumentation have so far largely limited learning experiments with real robots to well controlled lab settings. In this paper, we demonstrate how a small bipedal robot can autonomously learn to walk with minimal human intervention and with minimal instrumentation of the environment. We employ data-efficient off-policy deep reinforcement learning to learn to walk end-to-end, directly on hardware, using rewards that are computed exclusively from proprioceptive sensing. To allow the robot to autonomously adapt its behaviour to its environment, we additionally provide the agent with raw RGB camera images as input. By deploying two robots in different geographic locations while sharing data in a distributed learning setup, we achieve higher throughput and greater diversity of the training data. Our learning experiments constitute a step towards the long-term vision of learning “in the wild” for legged robots, and, to our knowledge, represent the first demonstration of learning a deep neural network controller for bipedal locomotion directly on hardware.
APA
Bloesch, M., Humplik, J., Patraucean, V., Hafner, R., Haarnoja, T., Byravan, A., Siegel, N.Y., Tunyasuvunakool, S., Casarini, F., Batchelor, N., Romano, F., Saliceti, S., Riedmiller, M., Eslami, S.M.A. & Heess, N.. (2022). Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:1502-1511 Available from https://proceedings.mlr.press/v164/bloesch22a.html.

Related Material