IMAGINATION POLICY: Using Generative Point Cloud Models for Learning Manipulation Policies

Haojie Huang, Karl Schmeckpeper, Dian Wang, Ondrej Biza, Yaoyao Qian, Haotian Liu, Mingxi Jia, Robert Platt, Robin Walters
Proceedings of The 8th Conference on Robot Learning, PMLR 270:5150-5165, 2025.

Abstract

Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose IMAGINATION POLICY, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, IMAGINATION POLICY generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-huang25h, title = {IMAGINATION POLICY: Using Generative Point Cloud Models for Learning Manipulation Policies}, author = {Huang, Haojie and Schmeckpeper, Karl and Wang, Dian and Biza, Ondrej and Qian, Yaoyao and Liu, Haotian and Jia, Mingxi and Platt, Robert and Walters, Robin}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {5150--5165}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/huang25h/huang25h.pdf}, url = {https://proceedings.mlr.press/v270/huang25h.html}, abstract = {Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose IMAGINATION POLICY, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, IMAGINATION POLICY generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.} }
Endnote
%0 Conference Paper %T IMAGINATION POLICY: Using Generative Point Cloud Models for Learning Manipulation Policies %A Haojie Huang %A Karl Schmeckpeper %A Dian Wang %A Ondrej Biza %A Yaoyao Qian %A Haotian Liu %A Mingxi Jia %A Robert Platt %A Robin Walters %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-huang25h %I PMLR %P 5150--5165 %U https://proceedings.mlr.press/v270/huang25h.html %V 270 %X Humans can imagine goal states during planning and perform actions to match those goals. In this work, we propose IMAGINATION POLICY, a novel multi-task key-frame policy network for solving high-precision pick and place tasks. Instead of learning actions directly, IMAGINATION POLICY generates point clouds to imagine desired states which are then translated to actions using rigid action estimation. This transforms action inference into a local generative task. We leverage pick and place symmetries underlying the tasks in the generation process and achieve extremely high sample efficiency and generalizability to unseen configurations. Finally, we demonstrate state-of-the-art performance across various tasks on the RLbench benchmark compared with several strong baselines and validate our approach on a real robot.
APA
Huang, H., Schmeckpeper, K., Wang, D., Biza, O., Qian, Y., Liu, H., Jia, M., Platt, R. & Walters, R.. (2025). IMAGINATION POLICY: Using Generative Point Cloud Models for Learning Manipulation Policies. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:5150-5165 Available from https://proceedings.mlr.press/v270/huang25h.html.

Related Material