Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR

Shahbaz P Qadri Syed, He Bai
Proceedings of the 7th Annual Learning for Dynamics \& Control Conference, PMLR 283:1378-1391, 2025.

Abstract

Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.

Cite this Paper

BibTeX
@InProceedings{pmlr-v283-syed25a, title = {Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR}, author = {Syed, Shahbaz P Qadri and Bai, He}, booktitle = {Proceedings of the 7th Annual Learning for Dynamics \& Control Conference}, pages = {1378--1391}, year = {2025}, editor = {Ozay, Necmiye and Balzano, Laura and Panagou, Dimitra and Abate, Alessandro}, volume = {283}, series = {Proceedings of Machine Learning Research}, month = {04--06 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v283/main/assets/syed25a/syed25a.pdf}, url = {https://proceedings.mlr.press/v283/syed25a.html}, abstract = {Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.} }
Endnote
%0 Conference Paper %T Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR %A Shahbaz P Qadri Syed %A He Bai %B Proceedings of the 7th Annual Learning for Dynamics \& Control Conference %C Proceedings of Machine Learning Research %D 2025 %E Necmiye Ozay %E Laura Balzano %E Dimitra Panagou %E Alessandro Abate %F pmlr-v283-syed25a %I PMLR %P 1378--1391 %U https://proceedings.mlr.press/v283/syed25a.html %V 283 %X Developing scalable and efficient reinforcement learning algorithms for cooperative multi-agent control has received significant attention over the past years. Existing literature has proposed inexact decompositions of local Q-functions based on empirical information structures between the agents. In this paper, we exploit inter-agent coupling information and propose a systematic approach to exactly decompose the local Q-function of each agent. We develop an approximate least square policy iteration algorithm based on the proposed decomposition and identify two architectures to learn the local Q-function for each agent. We establish that the worst-case sample complexity of the decomposition is equal to the centralized case and derive necessary and sufficient graphical conditions on the inter-agent couplings to achieve better sample efficiency. We demonstrate the improved sample efficiency and computational efficiency on numerical examples.
APA
Syed, S.P.Q. & Bai, H.. (2025). Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR. Proceedings of the 7th Annual Learning for Dynamics \& Control Conference, in Proceedings of Machine Learning Research 283:1378-1391 Available from https://proceedings.mlr.press/v283/syed25a.html.

Related Material