Improving Robustness via Risk Averse Distributional Reinforcement Learning

Rahul Singh, Qinsheng Zhang, Yongxin Chen
Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR 120:958-968, 2020.

Abstract

One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v120-singh20a, title = {Improving Robustness via Risk Averse Distributional Reinforcement Learning}, author = {Singh, Rahul and Zhang, Qinsheng and Chen, Yongxin}, booktitle = {Proceedings of the 2nd Conference on Learning for Dynamics and Control}, pages = {958--968}, year = {2020}, editor = {Bayen, Alexandre M. and Jadbabaie, Ali and Pappas, George and Parrilo, Pablo A. and Recht, Benjamin and Tomlin, Claire and Zeilinger, Melanie}, volume = {120}, series = {Proceedings of Machine Learning Research}, month = {10--11 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v120/singh20a/singh20a.pdf}, url = {https://proceedings.mlr.press/v120/singh20a.html}, abstract = {One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.} }
Endnote
%0 Conference Paper %T Improving Robustness via Risk Averse Distributional Reinforcement Learning %A Rahul Singh %A Qinsheng Zhang %A Yongxin Chen %B Proceedings of the 2nd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2020 %E Alexandre M. Bayen %E Ali Jadbabaie %E George Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire Tomlin %E Melanie Zeilinger %F pmlr-v120-singh20a %I PMLR %P 958--968 %U https://proceedings.mlr.press/v120/singh20a.html %V 120 %X One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.
APA
Singh, R., Zhang, Q. & Chen, Y.. (2020). Improving Robustness via Risk Averse Distributional Reinforcement Learning. Proceedings of the 2nd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 120:958-968 Available from https://proceedings.mlr.press/v120/singh20a.html.

Related Material