Robust Reinforcement Learning: A Constrained Game-theoretic Approach

Jing Yu, Clement Gehring, Florian Schäfer, Animashree Anandkumar
Proceedings of the 3rd Conference on Learning for Dynamics and Control, PMLR 144:1242-1254, 2021.

Abstract

Deep reinforcement learning (RL) methods provide state-of-art performance in complex control tasks. However, it has been widely recognized that RL methods often fail to generalize due to unaccounted uncertainties. In this work, we propose a game theoretic framework for robust reinforcement learning that comprises many previous works as special cases. We formulate robust RL as a constrained minimax game between the RL agent and an environmental agent which represents uncertainties such as model parameter variations and adversarial disturbances. To solve the competitive optimization problems arising in our framework, we propose to use competitive mirror descent (CMD). This method accounts for the interactive nature of the game at each iteration while using Bregman divergences to adapt to the global structure of the constraint set. We demonstrate an RRL policy gradient algorithm that leverages Lagrangian duality and CMD. We empirically show that our algorithm is stable for large step sizes, resulting in faster convergence on linear quadratic games.

Cite this Paper

BibTeX
@InProceedings{pmlr-v144-yu21a, title = {Robust Reinforcement Learning: A Constrained Game-theoretic Approach}, author = {Yu, Jing and Gehring, Clement and Sch\"afer, Florian and Anandkumar, Animashree}, booktitle = {Proceedings of the 3rd Conference on Learning for Dynamics and Control}, pages = {1242--1254}, year = {2021}, editor = {Jadbabaie, Ali and Lygeros, John and Pappas, George J. and A. Parrilo, Pablo and Recht, Benjamin and Tomlin, Claire J. and Zeilinger, Melanie N.}, volume = {144}, series = {Proceedings of Machine Learning Research}, month = {07 -- 08 June}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v144/yu21a/yu21a.pdf}, url = {https://proceedings.mlr.press/v144/yu21a.html}, abstract = {Deep reinforcement learning (RL) methods provide state-of-art performance in complex control tasks. However, it has been widely recognized that RL methods often fail to generalize due to unaccounted uncertainties. In this work, we propose a game theoretic framework for robust reinforcement learning that comprises many previous works as special cases. We formulate robust RL as a constrained minimax game between the RL agent and an environmental agent which represents uncertainties such as model parameter variations and adversarial disturbances. To solve the competitive optimization problems arising in our framework, we propose to use competitive mirror descent (CMD). This method accounts for the interactive nature of the game at each iteration while using Bregman divergences to adapt to the global structure of the constraint set. We demonstrate an RRL policy gradient algorithm that leverages Lagrangian duality and CMD. We empirically show that our algorithm is stable for large step sizes, resulting in faster convergence on linear quadratic games. } }
Endnote
%0 Conference Paper %T Robust Reinforcement Learning: A Constrained Game-theoretic Approach %A Jing Yu %A Clement Gehring %A Florian Schäfer %A Animashree Anandkumar %B Proceedings of the 3rd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2021 %E Ali Jadbabaie %E John Lygeros %E George J. Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire J. Tomlin %E Melanie N. Zeilinger %F pmlr-v144-yu21a %I PMLR %P 1242--1254 %U https://proceedings.mlr.press/v144/yu21a.html %V 144 %X Deep reinforcement learning (RL) methods provide state-of-art performance in complex control tasks. However, it has been widely recognized that RL methods often fail to generalize due to unaccounted uncertainties. In this work, we propose a game theoretic framework for robust reinforcement learning that comprises many previous works as special cases. We formulate robust RL as a constrained minimax game between the RL agent and an environmental agent which represents uncertainties such as model parameter variations and adversarial disturbances. To solve the competitive optimization problems arising in our framework, we propose to use competitive mirror descent (CMD). This method accounts for the interactive nature of the game at each iteration while using Bregman divergences to adapt to the global structure of the constraint set. We demonstrate an RRL policy gradient algorithm that leverages Lagrangian duality and CMD. We empirically show that our algorithm is stable for large step sizes, resulting in faster convergence on linear quadratic games.
APA
Yu, J., Gehring, C., Schäfer, F. & Anandkumar, A.. (2021). Robust Reinforcement Learning: A Constrained Game-theoretic Approach. Proceedings of the 3rd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 144:1242-1254 Available from https://proceedings.mlr.press/v144/yu21a.html.

Related Material