Re-understanding Finite-State Representations of Recurrent Policy Networks

Mohamad H Danesh, Anurag Koul, Alan Fern, Saeed Khorram
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2388-2397, 2021.

Abstract

We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine’s operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-danesh21a, title = {Re-understanding Finite-State Representations of Recurrent Policy Networks}, author = {Danesh, Mohamad H and Koul, Anurag and Fern, Alan and Khorram, Saeed}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2388--2397}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/danesh21a/danesh21a.pdf}, url = {https://proceedings.mlr.press/v139/danesh21a.html}, abstract = {We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine’s operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.} }
Endnote
%0 Conference Paper %T Re-understanding Finite-State Representations of Recurrent Policy Networks %A Mohamad H Danesh %A Anurag Koul %A Alan Fern %A Saeed Khorram %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-danesh21a %I PMLR %P 2388--2397 %U https://proceedings.mlr.press/v139/danesh21a.html %V 139 %X We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine’s operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.
APA
Danesh, M.H., Koul, A., Fern, A. & Khorram, S.. (2021). Re-understanding Finite-State Representations of Recurrent Policy Networks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2388-2397 Available from https://proceedings.mlr.press/v139/danesh21a.html.

Related Material