Distilling Task-Level Coordination Policies for Generalizable Multi-Agent Cooperation

Zimo Zhai, Manjie Xu, Wei Liang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:153917-153936, 2026.

Abstract

Large language models have shown strong reasoning abilities and are increasingly explored as high-level coordinators for multi-agent systems. However, directly deploying LLMs for coordination remains challenging, as effective policies often fail to reliably emerge at the low-level control stage, and inference costs limit scalability. We propose SynCoord (Synthetic Coordination Distillation), a self-supervised pipeline that distills task-level decision-making for cooperation from high-capacity reasoning models into lightweight agent policies. Our approach does not rely on explicit supervision or handcrafted coordination rules. Instead, we define a set of task-level tool interfaces that constrain LLM interaction and enable the collection of interaction trajectories, which are then used to train compact coordinated policies. This distillation process transfers coordination behaviors that are difficult to elicit through prompting alone, while substantially reducing inference overhead at execution time. We evaluate our method on cooperative multi-agent benchmarks including Overcooked-AI and Level-Based Foraging (LBF), under varying team sizes and environment scales. Experimental results show that the distilled policies achieve success rates and execution efficiency comparable to reinforcement learning–based methods, while exhibiting fewer erroneous or redundant actions. Moreover, the learned task-level coordination policy generalizes effectively to unseen team compositions and larger layouts without retraining.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhai26c, title = {Distilling Task-Level Coordination Policies for Generalizable Multi-Agent Cooperation}, author = {Zhai, Zimo and Xu, Manjie and Liang, Wei}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {153917--153936}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhai26c/zhai26c.pdf}, url = {https://proceedings.mlr.press/v306/zhai26c.html}, abstract = {Large language models have shown strong reasoning abilities and are increasingly explored as high-level coordinators for multi-agent systems. However, directly deploying LLMs for coordination remains challenging, as effective policies often fail to reliably emerge at the low-level control stage, and inference costs limit scalability. We propose SynCoord (Synthetic Coordination Distillation), a self-supervised pipeline that distills task-level decision-making for cooperation from high-capacity reasoning models into lightweight agent policies. Our approach does not rely on explicit supervision or handcrafted coordination rules. Instead, we define a set of task-level tool interfaces that constrain LLM interaction and enable the collection of interaction trajectories, which are then used to train compact coordinated policies. This distillation process transfers coordination behaviors that are difficult to elicit through prompting alone, while substantially reducing inference overhead at execution time. We evaluate our method on cooperative multi-agent benchmarks including Overcooked-AI and Level-Based Foraging (LBF), under varying team sizes and environment scales. Experimental results show that the distilled policies achieve success rates and execution efficiency comparable to reinforcement learning–based methods, while exhibiting fewer erroneous or redundant actions. Moreover, the learned task-level coordination policy generalizes effectively to unseen team compositions and larger layouts without retraining.} }
Endnote
%0 Conference Paper %T Distilling Task-Level Coordination Policies for Generalizable Multi-Agent Cooperation %A Zimo Zhai %A Manjie Xu %A Wei Liang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhai26c %I PMLR %P 153917--153936 %U https://proceedings.mlr.press/v306/zhai26c.html %V 306 %X Large language models have shown strong reasoning abilities and are increasingly explored as high-level coordinators for multi-agent systems. However, directly deploying LLMs for coordination remains challenging, as effective policies often fail to reliably emerge at the low-level control stage, and inference costs limit scalability. We propose SynCoord (Synthetic Coordination Distillation), a self-supervised pipeline that distills task-level decision-making for cooperation from high-capacity reasoning models into lightweight agent policies. Our approach does not rely on explicit supervision or handcrafted coordination rules. Instead, we define a set of task-level tool interfaces that constrain LLM interaction and enable the collection of interaction trajectories, which are then used to train compact coordinated policies. This distillation process transfers coordination behaviors that are difficult to elicit through prompting alone, while substantially reducing inference overhead at execution time. We evaluate our method on cooperative multi-agent benchmarks including Overcooked-AI and Level-Based Foraging (LBF), under varying team sizes and environment scales. Experimental results show that the distilled policies achieve success rates and execution efficiency comparable to reinforcement learning–based methods, while exhibiting fewer erroneous or redundant actions. Moreover, the learned task-level coordination policy generalizes effectively to unseen team compositions and larger layouts without retraining.
APA
Zhai, Z., Xu, M. & Liang, W.. (2026). Distilling Task-Level Coordination Policies for Generalizable Multi-Agent Cooperation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:153917-153936 Available from https://proceedings.mlr.press/v306/zhai26c.html.

Related Material