Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Xutong Zhao, Yaqi Xie
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:2971-2979, 2025.

Abstract

Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents to achieve a common goal. A key challenge in MARL is credit assignment, which involves assessing each agent’s contribution to the shared reward. Given the diversity of tasks, agents may perform different types of coordination, with rewards attributed to diverse and often overlapping agent subsets. In this work, we formalize the credit assignment level as the number of agents cooperating to obtain a reward, and address scenarios with multiple coexisting levels. We introduce a multi-level advantage formulation that performs explicit counterfactual reasoning to infer credits across distinct levels. Our method, Multi-level Advantage Credit Assignment (MACA), captures agent contributions at multiple levels by integrating advantage functions that reason about individual, joint, and correlated actions. Utilizing an attention-based framework, MACA identifies correlated agent relationships and constructs multi-level advantages to guide policy learning. Comprehensive experiments on challenging Starcraft v1 & v2 tasks demonstrate MACA’s superior performance, underscoring its efficacy in complex credit assignment scenarios.

Cite this Paper


BibTeX
@InProceedings{pmlr-v258-zhao25c, title = {Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning}, author = {Zhao, Xutong and Xie, Yaqi}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {2971--2979}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/zhao25c/zhao25c.pdf}, url = {https://proceedings.mlr.press/v258/zhao25c.html}, abstract = {Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents to achieve a common goal. A key challenge in MARL is credit assignment, which involves assessing each agent’s contribution to the shared reward. Given the diversity of tasks, agents may perform different types of coordination, with rewards attributed to diverse and often overlapping agent subsets. In this work, we formalize the credit assignment level as the number of agents cooperating to obtain a reward, and address scenarios with multiple coexisting levels. We introduce a multi-level advantage formulation that performs explicit counterfactual reasoning to infer credits across distinct levels. Our method, Multi-level Advantage Credit Assignment (MACA), captures agent contributions at multiple levels by integrating advantage functions that reason about individual, joint, and correlated actions. Utilizing an attention-based framework, MACA identifies correlated agent relationships and constructs multi-level advantages to guide policy learning. Comprehensive experiments on challenging Starcraft v1 & v2 tasks demonstrate MACA’s superior performance, underscoring its efficacy in complex credit assignment scenarios.} }
Endnote
%0 Conference Paper %T Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning %A Xutong Zhao %A Yaqi Xie %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-zhao25c %I PMLR %P 2971--2979 %U https://proceedings.mlr.press/v258/zhao25c.html %V 258 %X Cooperative multi-agent reinforcement learning (MARL) aims to coordinate multiple agents to achieve a common goal. A key challenge in MARL is credit assignment, which involves assessing each agent’s contribution to the shared reward. Given the diversity of tasks, agents may perform different types of coordination, with rewards attributed to diverse and often overlapping agent subsets. In this work, we formalize the credit assignment level as the number of agents cooperating to obtain a reward, and address scenarios with multiple coexisting levels. We introduce a multi-level advantage formulation that performs explicit counterfactual reasoning to infer credits across distinct levels. Our method, Multi-level Advantage Credit Assignment (MACA), captures agent contributions at multiple levels by integrating advantage functions that reason about individual, joint, and correlated actions. Utilizing an attention-based framework, MACA identifies correlated agent relationships and constructs multi-level advantages to guide policy learning. Comprehensive experiments on challenging Starcraft v1 & v2 tasks demonstrate MACA’s superior performance, underscoring its efficacy in complex credit assignment scenarios.
APA
Zhao, X. & Xie, Y.. (2025). Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:2971-2979 Available from https://proceedings.mlr.press/v258/zhao25c.html.

Related Material