Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation

Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, Hamsa Balakrishnan
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:25817-25833, 2023.

Abstract

We consider the problem of multi-agent navigation and collision avoidance when observations are limited to the local neighborhood of each agent. We propose InforMARL, a novel architecture for multi-agent reinforcement learning (MARL) which uses local information intelligently to compute paths for all the agents in a decentralized manner. Specifically, InforMARL aggregates information about the local neighborhood of agents for both the actor and the critic using a graph neural network and can be used in conjunction with any standard MARL algorithm. We show that (1) in training, InforMARL has better sample efficiency and performance than baseline approaches, despite using less information, and (2) in testing, it scales well to environments with arbitrary numbers of agents and obstacles. We illustrate these results using four task environments, including one with predetermined goals for each agent, and one in which the agents collectively try to cover all goals.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-nayak23a, title = {Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation}, author = {Nayak, Siddharth and Choi, Kenneth and Ding, Wenqi and Dolan, Sydney and Gopalakrishnan, Karthik and Balakrishnan, Hamsa}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {25817--25833}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/nayak23a/nayak23a.pdf}, url = {https://proceedings.mlr.press/v202/nayak23a.html}, abstract = {We consider the problem of multi-agent navigation and collision avoidance when observations are limited to the local neighborhood of each agent. We propose InforMARL, a novel architecture for multi-agent reinforcement learning (MARL) which uses local information intelligently to compute paths for all the agents in a decentralized manner. Specifically, InforMARL aggregates information about the local neighborhood of agents for both the actor and the critic using a graph neural network and can be used in conjunction with any standard MARL algorithm. We show that (1) in training, InforMARL has better sample efficiency and performance than baseline approaches, despite using less information, and (2) in testing, it scales well to environments with arbitrary numbers of agents and obstacles. We illustrate these results using four task environments, including one with predetermined goals for each agent, and one in which the agents collectively try to cover all goals.} }
Endnote
%0 Conference Paper %T Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation %A Siddharth Nayak %A Kenneth Choi %A Wenqi Ding %A Sydney Dolan %A Karthik Gopalakrishnan %A Hamsa Balakrishnan %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-nayak23a %I PMLR %P 25817--25833 %U https://proceedings.mlr.press/v202/nayak23a.html %V 202 %X We consider the problem of multi-agent navigation and collision avoidance when observations are limited to the local neighborhood of each agent. We propose InforMARL, a novel architecture for multi-agent reinforcement learning (MARL) which uses local information intelligently to compute paths for all the agents in a decentralized manner. Specifically, InforMARL aggregates information about the local neighborhood of agents for both the actor and the critic using a graph neural network and can be used in conjunction with any standard MARL algorithm. We show that (1) in training, InforMARL has better sample efficiency and performance than baseline approaches, despite using less information, and (2) in testing, it scales well to environments with arbitrary numbers of agents and obstacles. We illustrate these results using four task environments, including one with predetermined goals for each agent, and one in which the agents collectively try to cover all goals.
APA
Nayak, S., Choi, K., Ding, W., Dolan, S., Gopalakrishnan, K. & Balakrishnan, H.. (2023). Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:25817-25833 Available from https://proceedings.mlr.press/v202/nayak23a.html.

Related Material