TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning

Weiwei Liu, Wei Jing, lingping Gao, Ke Guo, Gang Xu, Yong Liu
Proceedings of The 7th Conference on Robot Learning, PMLR 229:2465-2477, 2023.

Abstract

Multi-agent reinforcement learning (MARL) has emerged as a popular technique in diverse domains due to its ability to automate system controller design and facilitate continuous intelligence learning. For instance, traffic flow is often trained with MARL to enable intelligent simulations for autonomous driving. However, The existing MARL algorithm only characterizes the relative degree of each agent’s contribution to the team, and cannot express the contribution that the team needs from the agent. Especially in the field of autonomous driving, the team changes over time, and the agent needs to act directly according to the needs of the team. To address these limitations, we propose an innovative method inspired by realistic traffic coordinators called the Traffic Coordinator Network (TraCo). Our approach leverages a combination of cross-attention and counterfactual advantage function, allowing us to extract distinctive characteristics of domain agents and accurately quantify the contribution that a team needs from an agent. Through experiments conducted on four traffic tasks, we demonstrate that our method outperforms existing approaches, yielding superior performance. Furthermore, our approach enables the emergence of rich and diverse social behaviors among vehicles within the traffic flow.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-liu23f, title = {TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning}, author = {Liu, Weiwei and Jing, Wei and Gao, lingping and Guo, Ke and Xu, Gang and Liu, Yong}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {2465--2477}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/liu23f/liu23f.pdf}, url = {https://proceedings.mlr.press/v229/liu23f.html}, abstract = {Multi-agent reinforcement learning (MARL) has emerged as a popular technique in diverse domains due to its ability to automate system controller design and facilitate continuous intelligence learning. For instance, traffic flow is often trained with MARL to enable intelligent simulations for autonomous driving. However, The existing MARL algorithm only characterizes the relative degree of each agent’s contribution to the team, and cannot express the contribution that the team needs from the agent. Especially in the field of autonomous driving, the team changes over time, and the agent needs to act directly according to the needs of the team. To address these limitations, we propose an innovative method inspired by realistic traffic coordinators called the Traffic Coordinator Network (TraCo). Our approach leverages a combination of cross-attention and counterfactual advantage function, allowing us to extract distinctive characteristics of domain agents and accurately quantify the contribution that a team needs from an agent. Through experiments conducted on four traffic tasks, we demonstrate that our method outperforms existing approaches, yielding superior performance. Furthermore, our approach enables the emergence of rich and diverse social behaviors among vehicles within the traffic flow.} }
Endnote
%0 Conference Paper %T TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning %A Weiwei Liu %A Wei Jing %A lingping Gao %A Ke Guo %A Gang Xu %A Yong Liu %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-liu23f %I PMLR %P 2465--2477 %U https://proceedings.mlr.press/v229/liu23f.html %V 229 %X Multi-agent reinforcement learning (MARL) has emerged as a popular technique in diverse domains due to its ability to automate system controller design and facilitate continuous intelligence learning. For instance, traffic flow is often trained with MARL to enable intelligent simulations for autonomous driving. However, The existing MARL algorithm only characterizes the relative degree of each agent’s contribution to the team, and cannot express the contribution that the team needs from the agent. Especially in the field of autonomous driving, the team changes over time, and the agent needs to act directly according to the needs of the team. To address these limitations, we propose an innovative method inspired by realistic traffic coordinators called the Traffic Coordinator Network (TraCo). Our approach leverages a combination of cross-attention and counterfactual advantage function, allowing us to extract distinctive characteristics of domain agents and accurately quantify the contribution that a team needs from an agent. Through experiments conducted on four traffic tasks, we demonstrate that our method outperforms existing approaches, yielding superior performance. Furthermore, our approach enables the emergence of rich and diverse social behaviors among vehicles within the traffic flow.
APA
Liu, W., Jing, W., Gao, l., Guo, K., Xu, G. & Liu, Y.. (2023). TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:2465-2477 Available from https://proceedings.mlr.press/v229/liu23f.html.

Related Material