Few-Shot In-Context Imitation Learning via Implicit Graph Alignment

Vitalis Vosylius, Edward Johns
Proceedings of The 7th Conference on Robot Learning, PMLR 229:3194-3213, 2023.

Abstract

Consider the following problem: given a few demonstrations of a task across a few different objects, how can a robot learn to perform that same task on new, previously unseen objects? This is challenging because the large variety of objects within a class makes it difficult to infer the task-relevant relationship between the new objects and the objects in the demonstrations. We address this by formulating imitation learning as a conditional alignment problem between graph representations of objects. Consequently, we show that this conditioning allows for in-context learning, where a robot can perform a task on a set of new objects immediately after the demonstrations, without any prior knowledge about the object class or any further training. In our experiments, we explore and validate our design choices, and we show that our method is highly effective for few-shot learning of several real-world, everyday tasks, whilst outperforming baselines. Videos are available on our project webpage at https://www.robot-learning.uk/implicit-graph-alignment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-vosylius23a, title = {Few-Shot In-Context Imitation Learning via Implicit Graph Alignment}, author = {Vosylius, Vitalis and Johns, Edward}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {3194--3213}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/vosylius23a/vosylius23a.pdf}, url = {https://proceedings.mlr.press/v229/vosylius23a.html}, abstract = {Consider the following problem: given a few demonstrations of a task across a few different objects, how can a robot learn to perform that same task on new, previously unseen objects? This is challenging because the large variety of objects within a class makes it difficult to infer the task-relevant relationship between the new objects and the objects in the demonstrations. We address this by formulating imitation learning as a conditional alignment problem between graph representations of objects. Consequently, we show that this conditioning allows for in-context learning, where a robot can perform a task on a set of new objects immediately after the demonstrations, without any prior knowledge about the object class or any further training. In our experiments, we explore and validate our design choices, and we show that our method is highly effective for few-shot learning of several real-world, everyday tasks, whilst outperforming baselines. Videos are available on our project webpage at https://www.robot-learning.uk/implicit-graph-alignment.} }
Endnote
%0 Conference Paper %T Few-Shot In-Context Imitation Learning via Implicit Graph Alignment %A Vitalis Vosylius %A Edward Johns %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-vosylius23a %I PMLR %P 3194--3213 %U https://proceedings.mlr.press/v229/vosylius23a.html %V 229 %X Consider the following problem: given a few demonstrations of a task across a few different objects, how can a robot learn to perform that same task on new, previously unseen objects? This is challenging because the large variety of objects within a class makes it difficult to infer the task-relevant relationship between the new objects and the objects in the demonstrations. We address this by formulating imitation learning as a conditional alignment problem between graph representations of objects. Consequently, we show that this conditioning allows for in-context learning, where a robot can perform a task on a set of new objects immediately after the demonstrations, without any prior knowledge about the object class or any further training. In our experiments, we explore and validate our design choices, and we show that our method is highly effective for few-shot learning of several real-world, everyday tasks, whilst outperforming baselines. Videos are available on our project webpage at https://www.robot-learning.uk/implicit-graph-alignment.
APA
Vosylius, V. & Johns, E.. (2023). Few-Shot In-Context Imitation Learning via Implicit Graph Alignment. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:3194-3213 Available from https://proceedings.mlr.press/v229/vosylius23a.html.

Related Material