Learning Intuitive Policies Using Action Features

Mingwei Ma, Jizhou Liu, Samuel Sokota, Max Kleiman-Weiner, Jakob Nicolaus Foerster
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:23358-23372, 2023.

Abstract

An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations. Humans take advantage of these relationships in highly intuitive ways. For instance, in the absence of a shared language, we might point to the object we desire or hold up our fingers to indicate how many objects we want. To address this challenge, we investigate the effect of network architecture on the propensity of learning algorithms to exploit these semantic relationships. Across a procedurally generated coordination task, we find that attention-based architectures that jointly process a featurized representation of observations and actions have a better inductive bias for learning intuitive policies. Through fine-grained evaluation and scenario analysis, we show that the resulting policies are human-interpretable. Moreover, such agents coordinate with people without training on any human data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-ma23e, title = {Learning Intuitive Policies Using Action Features}, author = {Ma, Mingwei and Liu, Jizhou and Sokota, Samuel and Kleiman-Weiner, Max and Foerster, Jakob Nicolaus}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {23358--23372}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/ma23e/ma23e.pdf}, url = {https://proceedings.mlr.press/v202/ma23e.html}, abstract = {An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations. Humans take advantage of these relationships in highly intuitive ways. For instance, in the absence of a shared language, we might point to the object we desire or hold up our fingers to indicate how many objects we want. To address this challenge, we investigate the effect of network architecture on the propensity of learning algorithms to exploit these semantic relationships. Across a procedurally generated coordination task, we find that attention-based architectures that jointly process a featurized representation of observations and actions have a better inductive bias for learning intuitive policies. Through fine-grained evaluation and scenario analysis, we show that the resulting policies are human-interpretable. Moreover, such agents coordinate with people without training on any human data.} }
Endnote
%0 Conference Paper %T Learning Intuitive Policies Using Action Features %A Mingwei Ma %A Jizhou Liu %A Samuel Sokota %A Max Kleiman-Weiner %A Jakob Nicolaus Foerster %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-ma23e %I PMLR %P 23358--23372 %U https://proceedings.mlr.press/v202/ma23e.html %V 202 %X An unaddressed challenge in multi-agent coordination is to enable AI agents to exploit the semantic relationships between the features of actions and the features of observations. Humans take advantage of these relationships in highly intuitive ways. For instance, in the absence of a shared language, we might point to the object we desire or hold up our fingers to indicate how many objects we want. To address this challenge, we investigate the effect of network architecture on the propensity of learning algorithms to exploit these semantic relationships. Across a procedurally generated coordination task, we find that attention-based architectures that jointly process a featurized representation of observations and actions have a better inductive bias for learning intuitive policies. Through fine-grained evaluation and scenario analysis, we show that the resulting policies are human-interpretable. Moreover, such agents coordinate with people without training on any human data.
APA
Ma, M., Liu, J., Sokota, S., Kleiman-Weiner, M. & Foerster, J.N.. (2023). Learning Intuitive Policies Using Action Features. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:23358-23372 Available from https://proceedings.mlr.press/v202/ma23e.html.

Related Material