Robust cooperative multi-agent reinforcement learning: A mean-field type game perspective

Muhammad Aneeq Uz Zaman, Mathieu Laurière, Alec Koppel, Tamer Başar
Proceedings of the 6th Annual Learning for Dynamics & Control Conference, PMLR 242:770-783, 2024.

Abstract

In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of stochastic and non-stochastic uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem as a worst-case (minimax) framework. Since this problem is intractable in general, we focus on the Linear Quadratic setting to enable derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.

Cite this Paper

BibTeX
@InProceedings{pmlr-v242-zaman24a, title = {Robust cooperative multi-agent reinforcement learning: {A} mean-field type game perspective}, author = {Zaman, Muhammad Aneeq Uz and Lauri\`{e}re, Mathieu and Koppel, Alec and Ba\c{s}ar, Tamer}, booktitle = {Proceedings of the 6th Annual Learning for Dynamics & Control Conference}, pages = {770--783}, year = {2024}, editor = {Abate, Alessandro and Cannon, Mark and Margellos, Kostas and Papachristodoulou, Antonis}, volume = {242}, series = {Proceedings of Machine Learning Research}, month = {15--17 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v242/zaman24a/zaman24a.pdf}, url = {https://proceedings.mlr.press/v242/zaman24a.html}, abstract = {In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of stochastic and non-stochastic uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem as a worst-case (minimax) framework. Since this problem is intractable in general, we focus on the Linear Quadratic setting to enable derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.} }
Endnote
%0 Conference Paper %T Robust cooperative multi-agent reinforcement learning: A mean-field type game perspective %A Muhammad Aneeq Uz Zaman %A Mathieu Laurière %A Alec Koppel %A Tamer Başar %B Proceedings of the 6th Annual Learning for Dynamics & Control Conference %C Proceedings of Machine Learning Research %D 2024 %E Alessandro Abate %E Mark Cannon %E Kostas Margellos %E Antonis Papachristodoulou %F pmlr-v242-zaman24a %I PMLR %P 770--783 %U https://proceedings.mlr.press/v242/zaman24a.html %V 242 %X In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of stochastic and non-stochastic uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem as a worst-case (minimax) framework. Since this problem is intractable in general, we focus on the Linear Quadratic setting to enable derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.
APA
Zaman, M.A.U., Laurière, M., Koppel, A. & Başar, T.. (2024). Robust cooperative multi-agent reinforcement learning: A mean-field type game perspective. Proceedings of the 6th Annual Learning for Dynamics & Control Conference, in Proceedings of Machine Learning Research 242:770-783 Available from https://proceedings.mlr.press/v242/zaman24a.html.

Related Material