Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation

Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1640-1651, 2023.

Abstract

Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on short-horizon tasks or make strong assumptions that full-state information is available, which prevents their use on deformable objects. In this paper, we propose PlAnning with Spatial-Temporal Abstraction (PASTA), which incorporates both spatial abstraction (reasoning about objects and their relations to each other) and temporal abstraction (reasoning over skills instead of low-level actions). Our framework maps high-dimension 3D observations such as point clouds into a set of latent vectors and plans over skill sequences on top of the latent set representation. We show that our method can effectively perform challenging sequential deformable object manipulation tasks in the real world, which require combining multiple tool-use skills such as cutting with a knife, pushing with a pusher, and spreading dough with a roller. Additional materials can be found at our project website: https://sites.google.com/view/pasta-plan.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-lin23b, title = {Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation}, author = {Lin, Xingyu and Qi, Carl and Zhang, Yunchu and Huang, Zhiao and Fragkiadaki, Katerina and Li, Yunzhu and Gan, Chuang and Held, David}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1640--1651}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/lin23b/lin23b.pdf}, url = {https://proceedings.mlr.press/v205/lin23b.html}, abstract = {Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on short-horizon tasks or make strong assumptions that full-state information is available, which prevents their use on deformable objects. In this paper, we propose PlAnning with Spatial-Temporal Abstraction (PASTA), which incorporates both spatial abstraction (reasoning about objects and their relations to each other) and temporal abstraction (reasoning over skills instead of low-level actions). Our framework maps high-dimension 3D observations such as point clouds into a set of latent vectors and plans over skill sequences on top of the latent set representation. We show that our method can effectively perform challenging sequential deformable object manipulation tasks in the real world, which require combining multiple tool-use skills such as cutting with a knife, pushing with a pusher, and spreading dough with a roller. Additional materials can be found at our project website: https://sites.google.com/view/pasta-plan.} }
Endnote
%0 Conference Paper %T Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation %A Xingyu Lin %A Carl Qi %A Yunchu Zhang %A Zhiao Huang %A Katerina Fragkiadaki %A Yunzhu Li %A Chuang Gan %A David Held %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-lin23b %I PMLR %P 1640--1651 %U https://proceedings.mlr.press/v205/lin23b.html %V 205 %X Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on short-horizon tasks or make strong assumptions that full-state information is available, which prevents their use on deformable objects. In this paper, we propose PlAnning with Spatial-Temporal Abstraction (PASTA), which incorporates both spatial abstraction (reasoning about objects and their relations to each other) and temporal abstraction (reasoning over skills instead of low-level actions). Our framework maps high-dimension 3D observations such as point clouds into a set of latent vectors and plans over skill sequences on top of the latent set representation. We show that our method can effectively perform challenging sequential deformable object manipulation tasks in the real world, which require combining multiple tool-use skills such as cutting with a knife, pushing with a pusher, and spreading dough with a roller. Additional materials can be found at our project website: https://sites.google.com/view/pasta-plan.
APA
Lin, X., Qi, C., Zhang, Y., Huang, Z., Fragkiadaki, K., Li, Y., Gan, C. & Held, D.. (2023). Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1640-1651 Available from https://proceedings.mlr.press/v205/lin23b.html.

Related Material