Concept Learning for Interpretable Multi-Agent Reinforcement Learning

Renos Zabounidis, Joseph Campbell, Simon Stepputtis, Dana Hughes, Katia P. Sycara
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1828-1837, 2023.

Abstract

Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-zabounidis23a, title = {Concept Learning for Interpretable Multi-Agent Reinforcement Learning}, author = {Zabounidis, Renos and Campbell, Joseph and Stepputtis, Simon and Hughes, Dana and Sycara, Katia P.}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1828--1837}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/zabounidis23a/zabounidis23a.pdf}, url = {https://proceedings.mlr.press/v205/zabounidis23a.html}, abstract = {Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.} }
Endnote
%0 Conference Paper %T Concept Learning for Interpretable Multi-Agent Reinforcement Learning %A Renos Zabounidis %A Joseph Campbell %A Simon Stepputtis %A Dana Hughes %A Katia P. Sycara %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-zabounidis23a %I PMLR %P 1828--1837 %U https://proceedings.mlr.press/v205/zabounidis23a.html %V 205 %X Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.
APA
Zabounidis, R., Campbell, J., Stepputtis, S., Hughes, D. & Sycara, K.P.. (2023). Concept Learning for Interpretable Multi-Agent Reinforcement Learning. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1828-1837 Available from https://proceedings.mlr.press/v205/zabounidis23a.html.

Related Material