Addressing Function Approximation Error in Actor-Critic Methods

Scott Fujimoto, Herke Hoof, David Meger
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1587-1596, 2018.

Abstract

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-fujimoto18a, title = {Addressing Function Approximation Error in Actor-Critic Methods}, author = {Fujimoto, Scott and van Hoof, Herke and Meger, David}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1587--1596}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/fujimoto18a/fujimoto18a.pdf}, url = {https://proceedings.mlr.press/v80/fujimoto18a.html}, abstract = {In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.} }
Endnote
%0 Conference Paper %T Addressing Function Approximation Error in Actor-Critic Methods %A Scott Fujimoto %A Herke Hoof %A David Meger %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-fujimoto18a %I PMLR %P 1587--1596 %U https://proceedings.mlr.press/v80/fujimoto18a.html %V 80 %X In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.
APA
Fujimoto, S., Hoof, H. & Meger, D.. (2018). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:1587-1596 Available from https://proceedings.mlr.press/v80/fujimoto18a.html.

Related Material