The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation

Mark Rowland, Yunhao Tang, Clare Lyle, Remi Munos, Marc G Bellemare, Will Dabney
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:29210-29231, 2023.

Abstract

We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-rowland23a, title = {The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation}, author = {Rowland, Mark and Tang, Yunhao and Lyle, Clare and Munos, Remi and Bellemare, Marc G and Dabney, Will}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {29210--29231}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/rowland23a/rowland23a.pdf}, url = {https://proceedings.mlr.press/v202/rowland23a.html}, abstract = {We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.} }
Endnote
%0 Conference Paper %T The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation %A Mark Rowland %A Yunhao Tang %A Clare Lyle %A Remi Munos %A Marc G Bellemare %A Will Dabney %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-rowland23a %I PMLR %P 29210--29231 %U https://proceedings.mlr.press/v202/rowland23a.html %V 202 %X We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.
APA
Rowland, M., Tang, Y., Lyle, C., Munos, R., Bellemare, M.G. & Dabney, W.. (2023). The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:29210-29231 Available from https://proceedings.mlr.press/v202/rowland23a.html.

Related Material