Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

Oron Anschel, Nir Baram, Nahum Shimkin
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:176-185, 2017.

Abstract

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-anschel17a, title = {Averaged-{DQN}: Variance Reduction and Stabilization for Deep Reinforcement Learning}, author = {Oron Anschel and Nir Baram and Nahum Shimkin}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {176--185}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/anschel17a/anschel17a.pdf}, url = {https://proceedings.mlr.press/v70/anschel17a.html}, abstract = {Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.} }
Endnote
%0 Conference Paper %T Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning %A Oron Anschel %A Nir Baram %A Nahum Shimkin %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-anschel17a %I PMLR %P 176--185 %U https://proceedings.mlr.press/v70/anschel17a.html %V 70 %X Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.
APA
Anschel, O., Baram, N. & Shimkin, N.. (2017). Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:176-185 Available from https://proceedings.mlr.press/v70/anschel17a.html.

Related Material