MELD: Meta-Reinforcement Learning from Images via Latent State Models

Zihao Zhao, Anusha Nagabandi, Kate Rakelly, Chelsea Finn, Sergey Levine
Proceedings of the 2020 Conference on Robot Learning, PMLR 155:1246-1261, 2021.

Abstract

Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks. However, the onerous data requirements of meta-training compounded with the challenge of learning from sensory inputs such as images have made meta-RL challenging to apply to real robotic systems. Latent state models, which learn compact state representations from a sequence of observations, can accelerate representation learning from visual inputs. In this paper, we leverage the perspective of meta-learning as task inference to show that latent state models can {\em also} perform meta-learning given an appropriately defined observation space. Building on this insight, we develop meta-RL with latent dynamics (MELD), an algorithm for meta-RL from images that performs inference in a latent state model to quickly acquire new skills given observations and rewards. MELD outperforms prior meta-RL methods on several simulated image-based robotic control problems, and enables a real WidowX robotic arm to insert an Ethernet cable into new locations given a sparse task completion signal after only $8$ hours of real world meta-training. To our knowledge, MELD is the first meta-RL algorithm trained in a real-world robotic control setting from images.

Cite this Paper

BibTeX
@InProceedings{pmlr-v155-zhao21d, title = {MELD: Meta-Reinforcement Learning from Images via Latent State Models}, author = {Zhao, Zihao and Nagabandi, Anusha and Rakelly, Kate and Finn, Chelsea and Levine, Sergey}, booktitle = {Proceedings of the 2020 Conference on Robot Learning}, pages = {1246--1261}, year = {2021}, editor = {Kober, Jens and Ramos, Fabio and Tomlin, Claire}, volume = {155}, series = {Proceedings of Machine Learning Research}, month = {16--18 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v155/zhao21d/zhao21d.pdf}, url = {https://proceedings.mlr.press/v155/zhao21d.html}, abstract = {Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks. However, the onerous data requirements of meta-training compounded with the challenge of learning from sensory inputs such as images have made meta-RL challenging to apply to real robotic systems. Latent state models, which learn compact state representations from a sequence of observations, can accelerate representation learning from visual inputs. In this paper, we leverage the perspective of meta-learning as task inference to show that latent state models can {\em also} perform meta-learning given an appropriately defined observation space. Building on this insight, we develop meta-RL with latent dynamics (MELD), an algorithm for meta-RL from images that performs inference in a latent state model to quickly acquire new skills given observations and rewards. MELD outperforms prior meta-RL methods on several simulated image-based robotic control problems, and enables a real WidowX robotic arm to insert an Ethernet cable into new locations given a sparse task completion signal after only $8$ hours of real world meta-training. To our knowledge, MELD is the first meta-RL algorithm trained in a real-world robotic control setting from images.} }
Endnote
%0 Conference Paper %T MELD: Meta-Reinforcement Learning from Images via Latent State Models %A Zihao Zhao %A Anusha Nagabandi %A Kate Rakelly %A Chelsea Finn %A Sergey Levine %B Proceedings of the 2020 Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2021 %E Jens Kober %E Fabio Ramos %E Claire Tomlin %F pmlr-v155-zhao21d %I PMLR %P 1246--1261 %U https://proceedings.mlr.press/v155/zhao21d.html %V 155 %X Meta-reinforcement learning algorithms can enable autonomous agents, such as robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks. However, the onerous data requirements of meta-training compounded with the challenge of learning from sensory inputs such as images have made meta-RL challenging to apply to real robotic systems. Latent state models, which learn compact state representations from a sequence of observations, can accelerate representation learning from visual inputs. In this paper, we leverage the perspective of meta-learning as task inference to show that latent state models can {\em also} perform meta-learning given an appropriately defined observation space. Building on this insight, we develop meta-RL with latent dynamics (MELD), an algorithm for meta-RL from images that performs inference in a latent state model to quickly acquire new skills given observations and rewards. MELD outperforms prior meta-RL methods on several simulated image-based robotic control problems, and enables a real WidowX robotic arm to insert an Ethernet cable into new locations given a sparse task completion signal after only $8$ hours of real world meta-training. To our knowledge, MELD is the first meta-RL algorithm trained in a real-world robotic control setting from images.
APA
Zhao, Z., Nagabandi, A., Rakelly, K., Finn, C. & Levine, S.. (2021). MELD: Meta-Reinforcement Learning from Images via Latent State Models. Proceedings of the 2020 Conference on Robot Learning, in Proceedings of Machine Learning Research 155:1246-1261 Available from https://proceedings.mlr.press/v155/zhao21d.html.

Related Material