Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph

Utkarsh Aashu Mishra, Yongxin Chen, Danfei Xu
Proceedings of The 8th Conference on Robot Learning, PMLR 270:1461-1472, 2025.

Abstract

Learning to plan for multi-step, multi-manipulator tasks is notoriously difficult because of the large search space and the complex constraint satisfaction problems. We present Generative Factor Chaining (GFC), a composable generative model for planning. GFC represents a planning problem as a spatial-temporal factor graph, where nodes represent objects and robots in the scene, spatial factors capture the distributions of valid relationships among nodes, and temporal factors represent the distributions of skill transitions. Each factor is implemented as a modular diffusion model, which are composed during inference to generate feasible long-horizon plans through bi-directional message passing. We show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. More details can be found at: https://sites.google.com/view/generative-factor-chaining

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-mishra25a, title = {Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph}, author = {Mishra, Utkarsh Aashu and Chen, Yongxin and Xu, Danfei}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {1461--1472}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/mishra25a/mishra25a.pdf}, url = {https://proceedings.mlr.press/v270/mishra25a.html}, abstract = {Learning to plan for multi-step, multi-manipulator tasks is notoriously difficult because of the large search space and the complex constraint satisfaction problems. We present Generative Factor Chaining (GFC), a composable generative model for planning. GFC represents a planning problem as a spatial-temporal factor graph, where nodes represent objects and robots in the scene, spatial factors capture the distributions of valid relationships among nodes, and temporal factors represent the distributions of skill transitions. Each factor is implemented as a modular diffusion model, which are composed during inference to generate feasible long-horizon plans through bi-directional message passing. We show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. More details can be found at: https://sites.google.com/view/generative-factor-chaining} }
Endnote
%0 Conference Paper %T Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph %A Utkarsh Aashu Mishra %A Yongxin Chen %A Danfei Xu %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-mishra25a %I PMLR %P 1461--1472 %U https://proceedings.mlr.press/v270/mishra25a.html %V 270 %X Learning to plan for multi-step, multi-manipulator tasks is notoriously difficult because of the large search space and the complex constraint satisfaction problems. We present Generative Factor Chaining (GFC), a composable generative model for planning. GFC represents a planning problem as a spatial-temporal factor graph, where nodes represent objects and robots in the scene, spatial factors capture the distributions of valid relationships among nodes, and temporal factors represent the distributions of skill transitions. Each factor is implemented as a modular diffusion model, which are composed during inference to generate feasible long-horizon plans through bi-directional message passing. We show that GFC can solve complex bimanual manipulation tasks and exhibits strong generalization to unseen planning tasks with novel combinations of objects and constraints. More details can be found at: https://sites.google.com/view/generative-factor-chaining
APA
Mishra, U.A., Chen, Y. & Xu, D.. (2025). Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:1461-1472 Available from https://proceedings.mlr.press/v270/mishra25a.html.

Related Material