Improving Robustness via Risk Averse Distributional Reinforcement Learning

Rahul Singh, Qinsheng Zhang, Yongxin Chen
; Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR 120:958-968, 2020.

Abstract

One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.

Cite this Paper


BibTeX
@InProceedings{pmlr-v120-singh20a, title = {Improving Robustness via Risk Averse Distributional Reinforcement Learning}, author = {Singh, Rahul and Zhang, Qinsheng and Chen, Yongxin}, booktitle = {Proceedings of the 2nd Conference on Learning for Dynamics and Control}, pages = {958--968}, year = {2020}, editor = {Alexandre M. Bayen and Ali Jadbabaie and George Pappas and Pablo A. Parrilo and Benjamin Recht and Claire Tomlin and Melanie Zeilinger}, volume = {120}, series = {Proceedings of Machine Learning Research}, address = {The Cloud}, month = {10--11 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v120/singh20a/singh20a.pdf}, url = {http://proceedings.mlr.press/v120/singh20a.html}, abstract = {One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.} }
Endnote
%0 Conference Paper %T Improving Robustness via Risk Averse Distributional Reinforcement Learning %A Rahul Singh %A Qinsheng Zhang %A Yongxin Chen %B Proceedings of the 2nd Conference on Learning for Dynamics and Control %C Proceedings of Machine Learning Research %D 2020 %E Alexandre M. Bayen %E Ali Jadbabaie %E George Pappas %E Pablo A. Parrilo %E Benjamin Recht %E Claire Tomlin %E Melanie Zeilinger %F pmlr-v120-singh20a %I PMLR %J Proceedings of Machine Learning Research %P 958--968 %U http://proceedings.mlr.press %V 120 %W PMLR %X One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.
APA
Singh, R., Zhang, Q. & Chen, Y.. (2020). Improving Robustness via Risk Averse Distributional Reinforcement Learning. Proceedings of the 2nd Conference on Learning for Dynamics and Control, in PMLR 120:958-968

Related Material