SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields

Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, Pulkit Agrawal
Proceedings of The 6th Conference on Robot Learning, PMLR 205:835-846, 2023.

Abstract

We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-simeonov23a, title = {SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields}, author = {Simeonov, Anthony and Du, Yilun and Lin, Yen-Chen and Garcia, Alberto Rodriguez and Kaelbling, Leslie Pack and Lozano-P\'erez, Tom\'as and Agrawal, Pulkit}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {835--846}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/simeonov23a/simeonov23a.pdf}, url = {https://proceedings.mlr.press/v205/simeonov23a.html}, abstract = {We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.} }
Endnote
%0 Conference Paper %T SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields %A Anthony Simeonov %A Yilun Du %A Yen-Chen Lin %A Alberto Rodriguez Garcia %A Leslie Pack Kaelbling %A Tomás Lozano-Pérez %A Pulkit Agrawal %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-simeonov23a %I PMLR %P 835--846 %U https://proceedings.mlr.press/v205/simeonov23a.html %V 205 %X We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.
APA
Simeonov, A., Du, Y., Lin, Y., Garcia, A.R., Kaelbling, L.P., Lozano-Pérez, T. & Agrawal, P.. (2023). SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:835-846 Available from https://proceedings.mlr.press/v205/simeonov23a.html.

Related Material