Logistic Q-Learning

Joan Bas-Serrano, Sebastian Curi, Andreas Krause, Gergely Neu
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:3610-3618, 2021.

Abstract

We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The method is closely related to the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010), with the key difference that our method introduces a Q-function that enables efficient exact model-free implementation. The main feature of our algorithm (called QREPS) is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error. We provide a practical saddle-point optimization method for minimizing this loss function and provide an error-propagation analysis that relates the quality of the individual updates to the performance of the output policy. Finally, we demonstrate the effectiveness of our method on a range of benchmark problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-bas-serrano21a, title = { Logistic Q-Learning }, author = {Bas-Serrano, Joan and Curi, Sebastian and Krause, Andreas and Neu, Gergely}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {3610--3618}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/bas-serrano21a/bas-serrano21a.pdf}, url = {https://proceedings.mlr.press/v130/bas-serrano21a.html}, abstract = { We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The method is closely related to the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010), with the key difference that our method introduces a Q-function that enables efficient exact model-free implementation. The main feature of our algorithm (called QREPS) is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error. We provide a practical saddle-point optimization method for minimizing this loss function and provide an error-propagation analysis that relates the quality of the individual updates to the performance of the output policy. Finally, we demonstrate the effectiveness of our method on a range of benchmark problems. } }
Endnote
%0 Conference Paper %T Logistic Q-Learning %A Joan Bas-Serrano %A Sebastian Curi %A Andreas Krause %A Gergely Neu %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-bas-serrano21a %I PMLR %P 3610--3618 %U https://proceedings.mlr.press/v130/bas-serrano21a.html %V 130 %X We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The method is closely related to the classic Relative Entropy Policy Search (REPS) algorithm of Peters et al. (2010), with the key difference that our method introduces a Q-function that enables efficient exact model-free implementation. The main feature of our algorithm (called QREPS) is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error. We provide a practical saddle-point optimization method for minimizing this loss function and provide an error-propagation analysis that relates the quality of the individual updates to the performance of the output policy. Finally, we demonstrate the effectiveness of our method on a range of benchmark problems.
APA
Bas-Serrano, J., Curi, S., Krause, A. & Neu, G.. (2021). Logistic Q-Learning . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:3610-3618 Available from https://proceedings.mlr.press/v130/bas-serrano21a.html.

Related Material