Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement

Anthony Simeonov, Ankit Goyal, Lucas Manuelli, Yen-Chen Lin, Alina Sarmiento, Alberto Rodriguez Garcia, Pulkit Agrawal, Dieter Fox
Proceedings of The 7th Conference on Robot Learning, PMLR 229:2030-2069, 2023.

Abstract

We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of both scenes and objects, and is trained from demonstrations to operate directly on 3D point clouds. Our system overcomes challenges associated with the existence of many geometrically-similar rearrangement solutions for a given scene. By leveraging an iterative pose de-noising training procedure, we can fit multi-modal demonstration data and produce multi-modal outputs while remaining precise and accurate. We also show the advantages of conditioning on relevant local geometric features while ignoring irrelevant global structure that harms both generalization and precision. We demonstrate our approach on three distinct rearrangement tasks that require handling multi-modality and generalization over object shape and pose in both simulation and the real world. Project website, code, and videos: https://anthonysimeonov.github.io/rpdiff-multi-modal

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-simeonov23a, title = {Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement}, author = {Simeonov, Anthony and Goyal, Ankit and Manuelli, Lucas and Lin, Yen-Chen and Sarmiento, Alina and Garcia, Alberto Rodriguez and Agrawal, Pulkit and Fox, Dieter}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {2030--2069}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/simeonov23a/simeonov23a.pdf}, url = {https://proceedings.mlr.press/v229/simeonov23a.html}, abstract = {We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of both scenes and objects, and is trained from demonstrations to operate directly on 3D point clouds. Our system overcomes challenges associated with the existence of many geometrically-similar rearrangement solutions for a given scene. By leveraging an iterative pose de-noising training procedure, we can fit multi-modal demonstration data and produce multi-modal outputs while remaining precise and accurate. We also show the advantages of conditioning on relevant local geometric features while ignoring irrelevant global structure that harms both generalization and precision. We demonstrate our approach on three distinct rearrangement tasks that require handling multi-modality and generalization over object shape and pose in both simulation and the real world. Project website, code, and videos: https://anthonysimeonov.github.io/rpdiff-multi-modal} }
Endnote
%0 Conference Paper %T Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement %A Anthony Simeonov %A Ankit Goyal %A Lucas Manuelli %A Yen-Chen Lin %A Alina Sarmiento %A Alberto Rodriguez Garcia %A Pulkit Agrawal %A Dieter Fox %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-simeonov23a %I PMLR %P 2030--2069 %U https://proceedings.mlr.press/v229/simeonov23a.html %V 229 %X We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of both scenes and objects, and is trained from demonstrations to operate directly on 3D point clouds. Our system overcomes challenges associated with the existence of many geometrically-similar rearrangement solutions for a given scene. By leveraging an iterative pose de-noising training procedure, we can fit multi-modal demonstration data and produce multi-modal outputs while remaining precise and accurate. We also show the advantages of conditioning on relevant local geometric features while ignoring irrelevant global structure that harms both generalization and precision. We demonstrate our approach on three distinct rearrangement tasks that require handling multi-modality and generalization over object shape and pose in both simulation and the real world. Project website, code, and videos: https://anthonysimeonov.github.io/rpdiff-multi-modal
APA
Simeonov, A., Goyal, A., Manuelli, L., Lin, Y., Sarmiento, A., Garcia, A.R., Agrawal, P. & Fox, D.. (2023). Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:2030-2069 Available from https://proceedings.mlr.press/v229/simeonov23a.html.

Related Material