Zero-Shot Reinforcement Learning via Function Encoders

Tyler Ingebrand, Amy Zhang, Ufuk Topcu
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:21007-21019, 2024.

Abstract

Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-ingebrand24a, title = {Zero-Shot Reinforcement Learning via Function Encoders}, author = {Ingebrand, Tyler and Zhang, Amy and Topcu, Ufuk}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {21007--21019}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/ingebrand24a/ingebrand24a.pdf}, url = {https://proceedings.mlr.press/v235/ingebrand24a.html}, abstract = {Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.} }
Endnote
%0 Conference Paper %T Zero-Shot Reinforcement Learning via Function Encoders %A Tyler Ingebrand %A Amy Zhang %A Ufuk Topcu %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-ingebrand24a %I PMLR %P 21007--21019 %U https://proceedings.mlr.press/v235/ingebrand24a.html %V 235 %X Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.
APA
Ingebrand, T., Zhang, A. & Topcu, U.. (2024). Zero-Shot Reinforcement Learning via Function Encoders. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:21007-21019 Available from https://proceedings.mlr.press/v235/ingebrand24a.html.

Related Material