SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields
Proceedings of The 6th Conference on Robot Learning, PMLR 205:835-846, 2023.
We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.