ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

Tonghan Wang, Heng Dong, Victor Lesser, Chongjie Zhang
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:9876-9886, 2020.

Abstract

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-wang20f, title = {{ROMA}: Multi-Agent Reinforcement Learning with Emergent Roles}, author = {Wang, Tonghan and Dong, Heng and Lesser, Victor and Zhang, Chongjie}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {9876--9886}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/wang20f/wang20f.pdf}, url = {https://proceedings.mlr.press/v119/wang20f.html}, abstract = {The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.} }
Endnote
%0 Conference Paper %T ROMA: Multi-Agent Reinforcement Learning with Emergent Roles %A Tonghan Wang %A Heng Dong %A Victor Lesser %A Chongjie Zhang %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-wang20f %I PMLR %P 9876--9886 %U https://proceedings.mlr.press/v119/wang20f.html %V 119 %X The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.
APA
Wang, T., Dong, H., Lesser, V. & Zhang, C.. (2020). ROMA: Multi-Agent Reinforcement Learning with Emergent Roles. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:9876-9886 Available from https://proceedings.mlr.press/v119/wang20f.html.

Related Material