BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, Chelsea Finn
Proceedings of the 5th Conference on Robot Learning, PMLR 164:991-1002, 2022.

Abstract

In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-jang22a, title = {BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning}, author = {Jang, Eric and Irpan, Alex and Khansari, Mohi and Kappler, Daniel and Ebert, Frederik and Lynch, Corey and Levine, Sergey and Finn, Chelsea}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {991--1002}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/jang22a/jang22a.pdf}, url = {https://proceedings.mlr.press/v164/jang22a.html}, abstract = {In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.} }
Endnote
%0 Conference Paper %T BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning %A Eric Jang %A Alex Irpan %A Mohi Khansari %A Daniel Kappler %A Frederik Ebert %A Corey Lynch %A Sergey Levine %A Chelsea Finn %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-jang22a %I PMLR %P 991--1002 %U https://proceedings.mlr.press/v164/jang22a.html %V 164 %X In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.
APA
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S. & Finn, C.. (2022). BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:991-1002 Available from https://proceedings.mlr.press/v164/jang22a.html.

Related Material