Don’t Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning

Homer Rich Walke, Jonathan Heewon Yang, Albert Yu, Aviral Kumar, Jędrzej Orbik, Avi Singh, Sergey Levine
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1652-1662, 2023.

Abstract

Reinforcement learning (RL) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real-world robotic RL typically requires time consuming data collection and frequent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled setting in which they were learned. In this work, we study how these challenges of real-world robotic learning can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with a new task, our system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. Our empirical results demonstrate that incorporating prior data into robotic reinforcement learning enables autonomous learning, substantially improves sample-efficiency of learning, and enables better generalization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-walke23a, title = {Don’t Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning}, author = {Walke, Homer Rich and Yang, Jonathan Heewon and Yu, Albert and Kumar, Aviral and Orbik, J\k{e}drzej and Singh, Avi and Levine, Sergey}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1652--1662}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/walke23a/walke23a.pdf}, url = {https://proceedings.mlr.press/v205/walke23a.html}, abstract = {Reinforcement learning (RL) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real-world robotic RL typically requires time consuming data collection and frequent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled setting in which they were learned. In this work, we study how these challenges of real-world robotic learning can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with a new task, our system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. Our empirical results demonstrate that incorporating prior data into robotic reinforcement learning enables autonomous learning, substantially improves sample-efficiency of learning, and enables better generalization.} }
Endnote
%0 Conference Paper %T Don’t Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning %A Homer Rich Walke %A Jonathan Heewon Yang %A Albert Yu %A Aviral Kumar %A Jędrzej Orbik %A Avi Singh %A Sergey Levine %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-walke23a %I PMLR %P 1652--1662 %U https://proceedings.mlr.press/v205/walke23a.html %V 205 %X Reinforcement learning (RL) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real-world robotic RL typically requires time consuming data collection and frequent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled setting in which they were learned. In this work, we study how these challenges of real-world robotic learning can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with a new task, our system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. Our empirical results demonstrate that incorporating prior data into robotic reinforcement learning enables autonomous learning, substantially improves sample-efficiency of learning, and enables better generalization.
APA
Walke, H.R., Yang, J.H., Yu, A., Kumar, A., Orbik, J., Singh, A. & Levine, S.. (2023). Don’t Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1652-1662 Available from https://proceedings.mlr.press/v205/walke23a.html.

Related Material