Adversarial Learning of Distributional Reinforcement Learning

Yang Sui, Yukun Huang, Hongtu Zhu, Fan Zhou
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:32783-32796, 2023.

Abstract

Reinforcement learning (RL) has made significant advancements in artificial intelligence. However, its real-world applications are limited due to differences between simulated environments and the actual world. Consequently, it is crucial to systematically analyze how each component of the RL system can affect the final model performance. In this study, we propose an adversarial learning framework for distributional reinforcement learning, which adopts the concept of influence measure from the statistics community. This framework enables us to detect performance loss caused by either the internal policy structure or the external state observation. The proposed influence measure is based on information geometry and has desirable properties of invariance. We demonstrate that the influence measure is useful for three diagnostic tasks: identifying fragile states in trajectories, determining the instability of the policy architecture, and pinpointing anomalously sensitive policy parameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-sui23a, title = {Adversarial Learning of Distributional Reinforcement Learning}, author = {Sui, Yang and Huang, Yukun and Zhu, Hongtu and Zhou, Fan}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {32783--32796}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/sui23a/sui23a.pdf}, url = {https://proceedings.mlr.press/v202/sui23a.html}, abstract = {Reinforcement learning (RL) has made significant advancements in artificial intelligence. However, its real-world applications are limited due to differences between simulated environments and the actual world. Consequently, it is crucial to systematically analyze how each component of the RL system can affect the final model performance. In this study, we propose an adversarial learning framework for distributional reinforcement learning, which adopts the concept of influence measure from the statistics community. This framework enables us to detect performance loss caused by either the internal policy structure or the external state observation. The proposed influence measure is based on information geometry and has desirable properties of invariance. We demonstrate that the influence measure is useful for three diagnostic tasks: identifying fragile states in trajectories, determining the instability of the policy architecture, and pinpointing anomalously sensitive policy parameters.} }
Endnote
%0 Conference Paper %T Adversarial Learning of Distributional Reinforcement Learning %A Yang Sui %A Yukun Huang %A Hongtu Zhu %A Fan Zhou %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-sui23a %I PMLR %P 32783--32796 %U https://proceedings.mlr.press/v202/sui23a.html %V 202 %X Reinforcement learning (RL) has made significant advancements in artificial intelligence. However, its real-world applications are limited due to differences between simulated environments and the actual world. Consequently, it is crucial to systematically analyze how each component of the RL system can affect the final model performance. In this study, we propose an adversarial learning framework for distributional reinforcement learning, which adopts the concept of influence measure from the statistics community. This framework enables us to detect performance loss caused by either the internal policy structure or the external state observation. The proposed influence measure is based on information geometry and has desirable properties of invariance. We demonstrate that the influence measure is useful for three diagnostic tasks: identifying fragile states in trajectories, determining the instability of the policy architecture, and pinpointing anomalously sensitive policy parameters.
APA
Sui, Y., Huang, Y., Zhu, H. & Zhou, F.. (2023). Adversarial Learning of Distributional Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:32783-32796 Available from https://proceedings.mlr.press/v202/sui23a.html.

Related Material