Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement

Kallol Saha, Amber Li, Angela Rodriguez-Izquierdo, Lifan Yu, Ben Eisner, Maxim Likhachev, David Held
Proceedings of The 9th Conference on Robot Learning, PMLR 305:489-512, 2025.

Abstract

Multi-object rearrangement is a challenging task that requires robots to reason about a physical 3D scene and the effects of a sequence of actions. While traditional task planning methods are shown to be effective for long-horizon manipulation, they require discretizing the continuous state and action space into symbolic descriptions of objects, object relationships, and actions. Our proposed method is instead able to take in a partially-observed point cloud observation of an initial scene and plan to a goal-satisfying configuration, without needing to discretize the set of actions or object relationships. To enable this, we formulate the planning problem as an A* search over the space of possible point cloud rearrangements. We sample point cloud transformations from a learned, domain-specific prior and then search for a sequence of such point cloud transformations that leads from the initial state to a goal. We evaluate our method in terms of task planning success and task execution success on a real-world, multi-step table bussing environment and a simulation block stacking environment. We experimentally demonstrate that our method produces successful plans and outperforms a policy-learning approach; we also perform ablations that show the importance of search in our approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-saha25a, title = {Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement}, author = {Saha, Kallol and Li, Amber and Rodriguez-Izquierdo, Angela and Yu, Lifan and Eisner, Ben and Likhachev, Maxim and Held, David}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {489--512}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/saha25a/saha25a.pdf}, url = {https://proceedings.mlr.press/v305/saha25a.html}, abstract = {Multi-object rearrangement is a challenging task that requires robots to reason about a physical 3D scene and the effects of a sequence of actions. While traditional task planning methods are shown to be effective for long-horizon manipulation, they require discretizing the continuous state and action space into symbolic descriptions of objects, object relationships, and actions. Our proposed method is instead able to take in a partially-observed point cloud observation of an initial scene and plan to a goal-satisfying configuration, without needing to discretize the set of actions or object relationships. To enable this, we formulate the planning problem as an A* search over the space of possible point cloud rearrangements. We sample point cloud transformations from a learned, domain-specific prior and then search for a sequence of such point cloud transformations that leads from the initial state to a goal. We evaluate our method in terms of task planning success and task execution success on a real-world, multi-step table bussing environment and a simulation block stacking environment. We experimentally demonstrate that our method produces successful plans and outperforms a policy-learning approach; we also perform ablations that show the importance of search in our approach.} }
Endnote
%0 Conference Paper %T Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement %A Kallol Saha %A Amber Li %A Angela Rodriguez-Izquierdo %A Lifan Yu %A Ben Eisner %A Maxim Likhachev %A David Held %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-saha25a %I PMLR %P 489--512 %U https://proceedings.mlr.press/v305/saha25a.html %V 305 %X Multi-object rearrangement is a challenging task that requires robots to reason about a physical 3D scene and the effects of a sequence of actions. While traditional task planning methods are shown to be effective for long-horizon manipulation, they require discretizing the continuous state and action space into symbolic descriptions of objects, object relationships, and actions. Our proposed method is instead able to take in a partially-observed point cloud observation of an initial scene and plan to a goal-satisfying configuration, without needing to discretize the set of actions or object relationships. To enable this, we formulate the planning problem as an A* search over the space of possible point cloud rearrangements. We sample point cloud transformations from a learned, domain-specific prior and then search for a sequence of such point cloud transformations that leads from the initial state to a goal. We evaluate our method in terms of task planning success and task execution success on a real-world, multi-step table bussing environment and a simulation block stacking environment. We experimentally demonstrate that our method produces successful plans and outperforms a policy-learning approach; we also perform ablations that show the importance of search in our approach.
APA
Saha, K., Li, A., Rodriguez-Izquierdo, A., Yu, L., Eisner, B., Likhachev, M. & Held, D.. (2025). Planning from Point Clouds over Continuous Actions for Multi-object Rearrangement. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:489-512 Available from https://proceedings.mlr.press/v305/saha25a.html.

Related Material