Scalable spectral representations for multiagent reinforcement learning in network MDPs

Zhaolin Ren, Runyu Zhang, Bo Dai, Na Li
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:550-558, 2025.

Abstract

Network Markov Decision Processes (MDPs), which are the de-facto model for multi-agent control, pose a significant challenge to efficient learning caused by the exponential growth of the global state-action space with the number of agents. In this work, utilizing the exponential decay property of network dynamics, we first derive scalable spectral local representations for multiagent reinforcement learning in network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. Building on these local spectral representations, we design a scalable algorithmic framework for multiagent reinforcement learning in continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm. Empirically, we validate the effectiveness of our scalable representation-based approach on two benchmark problems, and demonstrate the advantages of our approach over generic function approximation approaches to representing the local $Q$-functions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-ren25a, title = {Scalable spectral representations for multiagent reinforcement learning in network MDPs}, author = {Ren, Zhaolin and Zhang, Runyu and Dai, Bo and Li, Na}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {550--558}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/ren25a/ren25a.pdf}, url = {https://proceedings.mlr.press/v258/ren25a.html}, abstract = {Network Markov Decision Processes (MDPs), which are the de-facto model for multi-agent control, pose a significant challenge to efficient learning caused by the exponential growth of the global state-action space with the number of agents. In this work, utilizing the exponential decay property of network dynamics, we first derive scalable spectral local representations for multiagent reinforcement learning in network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. Building on these local spectral representations, we design a scalable algorithmic framework for multiagent reinforcement learning in continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm. Empirically, we validate the effectiveness of our scalable representation-based approach on two benchmark problems, and demonstrate the advantages of our approach over generic function approximation approaches to representing the local $Q$-functions.} }
Endnote
%0 Conference Paper %T Scalable spectral representations for multiagent reinforcement learning in network MDPs %A Zhaolin Ren %A Runyu Zhang %A Bo Dai %A Na Li %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-ren25a %I PMLR %P 550--558 %U https://proceedings.mlr.press/v258/ren25a.html %V 258 %X Network Markov Decision Processes (MDPs), which are the de-facto model for multi-agent control, pose a significant challenge to efficient learning caused by the exponential growth of the global state-action space with the number of agents. In this work, utilizing the exponential decay property of network dynamics, we first derive scalable spectral local representations for multiagent reinforcement learning in network MDPs, which induces a network linear subspace for the local $Q$-function of each agent. Building on these local spectral representations, we design a scalable algorithmic framework for multiagent reinforcement learning in continuous state-action network MDPs, and provide end-to-end guarantees for the convergence of our algorithm. Empirically, we validate the effectiveness of our scalable representation-based approach on two benchmark problems, and demonstrate the advantages of our approach over generic function approximation approaches to representing the local $Q$-functions.
APA
Ren, Z., Zhang, R., Dai, B. & Li, N.. (2025). Scalable spectral representations for multiagent reinforcement learning in network MDPs. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:550-558 Available from https://proceedings.mlr.press/v258/ren25a.html.

Related Material