Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees

Kishan Panaganti Badrinath, Dileep Kalathil
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:511-520, 2021.

Abstract

This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDPs framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some benchmark problems from OpenAI Gym.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-badrinath21a, title = {Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees}, author = {Badrinath, Kishan Panaganti and Kalathil, Dileep}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {511--520}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/badrinath21a/badrinath21a.pdf}, url = {https://proceedings.mlr.press/v139/badrinath21a.html}, abstract = {This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDPs framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some benchmark problems from OpenAI Gym.} }
Endnote
%0 Conference Paper %T Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees %A Kishan Panaganti Badrinath %A Dileep Kalathil %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-badrinath21a %I PMLR %P 511--520 %U https://proceedings.mlr.press/v139/badrinath21a.html %V 139 %X This paper addresses the problem of model-free reinforcement learning for Robust Markov Decision Process (RMDP) with large state spaces. The goal of the RMDPs framework is to find a policy that is robust against the parameter uncertainties due to the mismatch between the simulator model and real-world settings. We first propose the Robust Least Squares Policy Evaluation algorithm, which is a multi-step online model-free learning algorithm for policy evaluation. We prove the convergence of this algorithm using stochastic approximation techniques. We then propose Robust Least Squares Policy Iteration (RLSPI) algorithm for learning the optimal robust policy. We also give a general weighted Euclidean norm bound on the error (closeness to optimality) of the resulting policy. Finally, we demonstrate the performance of our RLSPI algorithm on some benchmark problems from OpenAI Gym.
APA
Badrinath, K.P. & Kalathil, D.. (2021). Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:511-520 Available from https://proceedings.mlr.press/v139/badrinath21a.html.

Related Material