Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, Thore Graepel
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:6187-6199, 2021.

Abstract

Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent’s behavior constitutes (part of) another agent’s environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-leibo21a, title = {Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot}, author = {Leibo, Joel Z and Due{\~n}ez-Guzman, Edgar A and Vezhnevets, Alexander and Agapiou, John P and Sunehag, Peter and Koster, Raphael and Matyas, Jayd and Beattie, Charlie and Mordatch, Igor and Graepel, Thore}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {6187--6199}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/leibo21a/leibo21a.pdf}, url = {https://proceedings.mlr.press/v139/leibo21a.html}, abstract = {Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent’s behavior constitutes (part of) another agent’s environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.} }
Endnote
%0 Conference Paper %T Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot %A Joel Z Leibo %A Edgar A Dueñez-Guzman %A Alexander Vezhnevets %A John P Agapiou %A Peter Sunehag %A Raphael Koster %A Jayd Matyas %A Charlie Beattie %A Igor Mordatch %A Thore Graepel %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-leibo21a %I PMLR %P 6187--6199 %U https://proceedings.mlr.press/v139/leibo21a.html %V 139 %X Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent’s behavior constitutes (part of) another agent’s environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.
APA
Leibo, J.Z., Dueñez-Guzman, E.A., Vezhnevets, A., Agapiou, J.P., Sunehag, P., Koster, R., Matyas, J., Beattie, C., Mordatch, I. & Graepel, T.. (2021). Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6187-6199 Available from https://proceedings.mlr.press/v139/leibo21a.html.

Related Material