Auxiliary task discovery through generate-and-test

Banafsheh Rafiee, Sina Ghiassian, Jun Jin, Richard Sutton, Jun Luo, Adam White
Proceedings of The 2nd Conference on Lifelong Learning Agents, PMLR 232:703-714, 2023.

Abstract

In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks’ usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks and learning without auxiliary tasks across a suite of environments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v232-rafiee23a, title = {Auxiliary task discovery through generate-and-test}, author = {Rafiee, Banafsheh and Ghiassian, Sina and Jin, Jun and Sutton, Richard and Luo, Jun and White, Adam}, booktitle = {Proceedings of The 2nd Conference on Lifelong Learning Agents}, pages = {703--714}, year = {2023}, editor = {Chandar, Sarath and Pascanu, Razvan and Sedghi, Hanie and Precup, Doina}, volume = {232}, series = {Proceedings of Machine Learning Research}, month = {22--25 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v232/rafiee23a/rafiee23a.pdf}, url = {https://proceedings.mlr.press/v232/rafiee23a.html}, abstract = {In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks’ usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks and learning without auxiliary tasks across a suite of environments. } }
Endnote
%0 Conference Paper %T Auxiliary task discovery through generate-and-test %A Banafsheh Rafiee %A Sina Ghiassian %A Jun Jin %A Richard Sutton %A Jun Luo %A Adam White %B Proceedings of The 2nd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2023 %E Sarath Chandar %E Razvan Pascanu %E Hanie Sedghi %E Doina Precup %F pmlr-v232-rafiee23a %I PMLR %P 703--714 %U https://proceedings.mlr.press/v232/rafiee23a.html %V 232 %X In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks’ usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks and learning without auxiliary tasks across a suite of environments.
APA
Rafiee, B., Ghiassian, S., Jin, J., Sutton, R., Luo, J. & White, A.. (2023). Auxiliary task discovery through generate-and-test. Proceedings of The 2nd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 232:703-714 Available from https://proceedings.mlr.press/v232/rafiee23a.html.

Related Material