Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game AI

Lei Han, Peng Sun, Yali Du, Jiechao Xiong, Qing Wang, Xinghai Sun, Han Liu, Tong Zhang
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2576-2585, 2019.

Abstract

We consider the problem of multi-agent reinforcement learning (MARL) in video game AI, where the agents are located in a spatial grid-world environment and the number of agents varies both within and across episodes. The challenge is to flexibly control an arbitrary number of agents while achieving effective collaboration. Existing MARL methods usually suffer from the trade-off between these two considerations. To address the issue, we propose a novel architecture that learns a spatial joint representation of all the agents and outputs grid-wise actions. Each agent will be controlled independently by taking the action from the grid it occupies. By viewing the state information as a grid feature map, we employ a convolutional encoder-decoder as the policy network. This architecture naturally promotes agent communication because of the large receptive field provided by the stacked convolutional layers. Moreover, the spatially shared convolutional parameters enable fast parallel exploration that the experiences discovered by one agent can be immediately transferred to others. The proposed method can be conveniently integrated with general reinforcement learning algorithms, e.g., PPO and Q-learning. We demonstrate the effectiveness of the proposed method in extensive challenging multi-agent tasks in StarCraft II.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-han19a, title = {Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game {AI}}, author = {Han, Lei and Sun, Peng and Du, Yali and Xiong, Jiechao and Wang, Qing and Sun, Xinghai and Liu, Han and Zhang, Tong}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {2576--2585}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/han19a/han19a.pdf}, url = {https://proceedings.mlr.press/v97/han19a.html}, abstract = {We consider the problem of multi-agent reinforcement learning (MARL) in video game AI, where the agents are located in a spatial grid-world environment and the number of agents varies both within and across episodes. The challenge is to flexibly control an arbitrary number of agents while achieving effective collaboration. Existing MARL methods usually suffer from the trade-off between these two considerations. To address the issue, we propose a novel architecture that learns a spatial joint representation of all the agents and outputs grid-wise actions. Each agent will be controlled independently by taking the action from the grid it occupies. By viewing the state information as a grid feature map, we employ a convolutional encoder-decoder as the policy network. This architecture naturally promotes agent communication because of the large receptive field provided by the stacked convolutional layers. Moreover, the spatially shared convolutional parameters enable fast parallel exploration that the experiences discovered by one agent can be immediately transferred to others. The proposed method can be conveniently integrated with general reinforcement learning algorithms, e.g., PPO and Q-learning. We demonstrate the effectiveness of the proposed method in extensive challenging multi-agent tasks in StarCraft II.} }
Endnote
%0 Conference Paper %T Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game AI %A Lei Han %A Peng Sun %A Yali Du %A Jiechao Xiong %A Qing Wang %A Xinghai Sun %A Han Liu %A Tong Zhang %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-han19a %I PMLR %P 2576--2585 %U https://proceedings.mlr.press/v97/han19a.html %V 97 %X We consider the problem of multi-agent reinforcement learning (MARL) in video game AI, where the agents are located in a spatial grid-world environment and the number of agents varies both within and across episodes. The challenge is to flexibly control an arbitrary number of agents while achieving effective collaboration. Existing MARL methods usually suffer from the trade-off between these two considerations. To address the issue, we propose a novel architecture that learns a spatial joint representation of all the agents and outputs grid-wise actions. Each agent will be controlled independently by taking the action from the grid it occupies. By viewing the state information as a grid feature map, we employ a convolutional encoder-decoder as the policy network. This architecture naturally promotes agent communication because of the large receptive field provided by the stacked convolutional layers. Moreover, the spatially shared convolutional parameters enable fast parallel exploration that the experiences discovered by one agent can be immediately transferred to others. The proposed method can be conveniently integrated with general reinforcement learning algorithms, e.g., PPO and Q-learning. We demonstrate the effectiveness of the proposed method in extensive challenging multi-agent tasks in StarCraft II.
APA
Han, L., Sun, P., Du, Y., Xiong, J., Wang, Q., Sun, X., Liu, H. & Zhang, T.. (2019). Grid-Wise Control for Multi-Agent Reinforcement Learning in Video Game AI. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:2576-2585 Available from https://proceedings.mlr.press/v97/han19a.html.

Related Material