Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization

Yuan Zhang, Jianhong Wang, Joschka Boedecker
Proceedings of The 7th Conference on Robot Learning, PMLR 229:1400-1424, 2023.

Abstract

Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations, which excessively restricts its application for real-world robotics. Prior work claimed that adding regularization to the value function is equivalent to learning a robust policy under uncertain transitions. Although the regularization-robustness transformation is appealing for its simplicity and efficiency, it is still lacking in continuous control tasks. In this paper, we propose a new regularizer named Uncertainty Set Regularizer (USR), to formulate the uncertainty set on the parametric space of a transition function. To deal with unknown uncertainty sets, we further propose a novel adversarial approach to generate them based on the value function. We evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark and the Unitree A1 Robot, demonstrating improvements in the robust performance of perturbed testing environments and sim-to-real scenarios.

Cite this Paper

BibTeX
@InProceedings{pmlr-v229-zhang23d, title = {Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization}, author = {Zhang, Yuan and Wang, Jianhong and Boedecker, Joschka}, booktitle = {Proceedings of The 7th Conference on Robot Learning}, pages = {1400--1424}, year = {2023}, editor = {Tan, Jie and Toussaint, Marc and Darvish, Kourosh}, volume = {229}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v229/zhang23d/zhang23d.pdf}, url = {https://proceedings.mlr.press/v229/zhang23d.html}, abstract = {Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations, which excessively restricts its application for real-world robotics. Prior work claimed that adding regularization to the value function is equivalent to learning a robust policy under uncertain transitions. Although the regularization-robustness transformation is appealing for its simplicity and efficiency, it is still lacking in continuous control tasks. In this paper, we propose a new regularizer named Uncertainty Set Regularizer (USR), to formulate the uncertainty set on the parametric space of a transition function. To deal with unknown uncertainty sets, we further propose a novel adversarial approach to generate them based on the value function. We evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark and the Unitree A1 Robot, demonstrating improvements in the robust performance of perturbed testing environments and sim-to-real scenarios.} }
Endnote
%0 Conference Paper %T Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization %A Yuan Zhang %A Jianhong Wang %A Joschka Boedecker %B Proceedings of The 7th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Jie Tan %E Marc Toussaint %E Kourosh Darvish %F pmlr-v229-zhang23d %I PMLR %P 1400--1424 %U https://proceedings.mlr.press/v229/zhang23d.html %V 229 %X Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations, which excessively restricts its application for real-world robotics. Prior work claimed that adding regularization to the value function is equivalent to learning a robust policy under uncertain transitions. Although the regularization-robustness transformation is appealing for its simplicity and efficiency, it is still lacking in continuous control tasks. In this paper, we propose a new regularizer named Uncertainty Set Regularizer (USR), to formulate the uncertainty set on the parametric space of a transition function. To deal with unknown uncertainty sets, we further propose a novel adversarial approach to generate them based on the value function. We evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark and the Unitree A1 Robot, demonstrating improvements in the robust performance of perturbed testing environments and sim-to-real scenarios.
APA
Zhang, Y., Wang, J. & Boedecker, J.. (2023). Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization. Proceedings of The 7th Conference on Robot Learning, in Proceedings of Machine Learning Research 229:1400-1424 Available from https://proceedings.mlr.press/v229/zhang23d.html.

Related Material