Towards a better understanding of representation dynamics under TD-learning

Yunhao Tang, Remi Munos
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:33720-33738, 2023.

Abstract

TD-learning is a foundation reinforcement learning (RL) algorithm for value prediction. Critical to the accuracy of value predictions is the quality of state representations. In this work, we consider the question: how does end-to-end TD-learning impact the representation over time? Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning. We first show that when the environments are reversible, end-to-end TD-learning strictly decreases the value approximation error over time. Under further assumptions on the environments, we can connect the representation dynamics with spectral decomposition over the transition matrix. This latter finding establishes fitting multiple value functions from randomly generated rewards as a useful auxiliary task for representation learning, as we empirically validate on both tabular and Atari game suites.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-tang23g, title = {Towards a better understanding of representation dynamics under {TD}-learning}, author = {Tang, Yunhao and Munos, Remi}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {33720--33738}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/tang23g/tang23g.pdf}, url = {https://proceedings.mlr.press/v202/tang23g.html}, abstract = {TD-learning is a foundation reinforcement learning (RL) algorithm for value prediction. Critical to the accuracy of value predictions is the quality of state representations. In this work, we consider the question: how does end-to-end TD-learning impact the representation over time? Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning. We first show that when the environments are reversible, end-to-end TD-learning strictly decreases the value approximation error over time. Under further assumptions on the environments, we can connect the representation dynamics with spectral decomposition over the transition matrix. This latter finding establishes fitting multiple value functions from randomly generated rewards as a useful auxiliary task for representation learning, as we empirically validate on both tabular and Atari game suites.} }
Endnote
%0 Conference Paper %T Towards a better understanding of representation dynamics under TD-learning %A Yunhao Tang %A Remi Munos %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-tang23g %I PMLR %P 33720--33738 %U https://proceedings.mlr.press/v202/tang23g.html %V 202 %X TD-learning is a foundation reinforcement learning (RL) algorithm for value prediction. Critical to the accuracy of value predictions is the quality of state representations. In this work, we consider the question: how does end-to-end TD-learning impact the representation over time? Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning. We first show that when the environments are reversible, end-to-end TD-learning strictly decreases the value approximation error over time. Under further assumptions on the environments, we can connect the representation dynamics with spectral decomposition over the transition matrix. This latter finding establishes fitting multiple value functions from randomly generated rewards as a useful auxiliary task for representation learning, as we empirically validate on both tabular and Atari game suites.
APA
Tang, Y. & Munos, R.. (2023). Towards a better understanding of representation dynamics under TD-learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:33720-33738 Available from https://proceedings.mlr.press/v202/tang23g.html.

Related Material