BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images

Sandesh Adhikary, Anqi Li, Byron Boots
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:228-245, 2024.

Abstract

Training reinforcement learning (RL) agents directly from high-dimensional image observations continues to be a challenging problem. Recent line of work on behavioral distances proposes to learn representations that encode behavioral similarities quantified by the bisimulation metric. By learning an isometric mapping to a lower dimensional space that preserves this metric, such methods attempt to learn representations that group together functionally similar states. However, such an isometric mapping may not exist, making the learning objective ill-defined. We propose an alternative objective that allows distortions in long-range distances, while preserving local metric structure – inducing representations that highlight natural clusters in the state space. This leads to new representations, which we term Behavioral Eigenmaps (BeigeMaps), corresponding to the eigenfunctions of similarity kernels induced by behavioral distances. We empirically demonstrate that when added as a drop-in modification, BeigeMaps improve the policy performance of prior behavioral distance based RL algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-adhikary24a, title = {{B}eige{M}aps: Behavioral Eigenmaps for Reinforcement Learning from Images}, author = {Adhikary, Sandesh and Li, Anqi and Boots, Byron}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {228--245}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/adhikary24a/adhikary24a.pdf}, url = {https://proceedings.mlr.press/v235/adhikary24a.html}, abstract = {Training reinforcement learning (RL) agents directly from high-dimensional image observations continues to be a challenging problem. Recent line of work on behavioral distances proposes to learn representations that encode behavioral similarities quantified by the bisimulation metric. By learning an isometric mapping to a lower dimensional space that preserves this metric, such methods attempt to learn representations that group together functionally similar states. However, such an isometric mapping may not exist, making the learning objective ill-defined. We propose an alternative objective that allows distortions in long-range distances, while preserving local metric structure – inducing representations that highlight natural clusters in the state space. This leads to new representations, which we term Behavioral Eigenmaps (BeigeMaps), corresponding to the eigenfunctions of similarity kernels induced by behavioral distances. We empirically demonstrate that when added as a drop-in modification, BeigeMaps improve the policy performance of prior behavioral distance based RL algorithms.} }
Endnote
%0 Conference Paper %T BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images %A Sandesh Adhikary %A Anqi Li %A Byron Boots %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-adhikary24a %I PMLR %P 228--245 %U https://proceedings.mlr.press/v235/adhikary24a.html %V 235 %X Training reinforcement learning (RL) agents directly from high-dimensional image observations continues to be a challenging problem. Recent line of work on behavioral distances proposes to learn representations that encode behavioral similarities quantified by the bisimulation metric. By learning an isometric mapping to a lower dimensional space that preserves this metric, such methods attempt to learn representations that group together functionally similar states. However, such an isometric mapping may not exist, making the learning objective ill-defined. We propose an alternative objective that allows distortions in long-range distances, while preserving local metric structure – inducing representations that highlight natural clusters in the state space. This leads to new representations, which we term Behavioral Eigenmaps (BeigeMaps), corresponding to the eigenfunctions of similarity kernels induced by behavioral distances. We empirically demonstrate that when added as a drop-in modification, BeigeMaps improve the policy performance of prior behavioral distance based RL algorithms.
APA
Adhikary, S., Li, A. & Boots, B.. (2024). BeigeMaps: Behavioral Eigenmaps for Reinforcement Learning from Images. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:228-245 Available from https://proceedings.mlr.press/v235/adhikary24a.html.

Related Material