On the Effect of Auxiliary Tasks on Representation Dynamics

Clare Lyle, Mark Rowland, Georg Ostrovski, Will Dabney
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:1-9, 2021.

Abstract

While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-lyle21a, title = { On the Effect of Auxiliary Tasks on Representation Dynamics }, author = {Lyle, Clare and Rowland, Mark and Ostrovski, Georg and Dabney, Will}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {1--9}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/lyle21a/lyle21a.pdf}, url = {https://proceedings.mlr.press/v130/lyle21a.html}, abstract = { While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments. } }
Endnote
%0 Conference Paper %T On the Effect of Auxiliary Tasks on Representation Dynamics %A Clare Lyle %A Mark Rowland %A Georg Ostrovski %A Will Dabney %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-lyle21a %I PMLR %P 1--9 %U https://proceedings.mlr.press/v130/lyle21a.html %V 130 %X While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.
APA
Lyle, C., Rowland, M., Ostrovski, G. & Dabney, W.. (2021). On the Effect of Auxiliary Tasks on Representation Dynamics . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:1-9 Available from https://proceedings.mlr.press/v130/lyle21a.html.

Related Material