On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning

Hojoon Lee, Koanho Lee, Dongyoon Hwang, Hyunho Lee, Byungkun Lee, Jaegul Choo
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:18988-19009, 2023.

Abstract

Recently, unsupervised representation learning (URL) has improved the sample efficiency of Reinforcement Learning (RL) by pretraining a model from a large unlabeled dataset. The underlying principle of these methods is to learn temporally predictive representations by predicting future states in the latent space. However, an important challenge of this approach is the representational collapse, where the subspace of the latent representations collapses into a low-dimensional manifold. To address this issue, we propose a novel URL framework that causally predicts future states while increasing the dimension of the latent manifold by decorrelating the features in the latent space. Through extensive empirical studies, we demonstrate that our framework effectively learns predictive representations without collapse, which significantly improves the sample efficiency of state-of-the-art URL methods on the Atari 100k benchmark. The code is available at https://github.com/dojeon-ai/SimTPR.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-lee23l, title = {On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning}, author = {Lee, Hojoon and Lee, Koanho and Hwang, Dongyoon and Lee, Hyunho and Lee, Byungkun and Choo, Jaegul}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {18988--19009}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/lee23l/lee23l.pdf}, url = {https://proceedings.mlr.press/v202/lee23l.html}, abstract = {Recently, unsupervised representation learning (URL) has improved the sample efficiency of Reinforcement Learning (RL) by pretraining a model from a large unlabeled dataset. The underlying principle of these methods is to learn temporally predictive representations by predicting future states in the latent space. However, an important challenge of this approach is the representational collapse, where the subspace of the latent representations collapses into a low-dimensional manifold. To address this issue, we propose a novel URL framework that causally predicts future states while increasing the dimension of the latent manifold by decorrelating the features in the latent space. Through extensive empirical studies, we demonstrate that our framework effectively learns predictive representations without collapse, which significantly improves the sample efficiency of state-of-the-art URL methods on the Atari 100k benchmark. The code is available at https://github.com/dojeon-ai/SimTPR.} }
Endnote
%0 Conference Paper %T On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning %A Hojoon Lee %A Koanho Lee %A Dongyoon Hwang %A Hyunho Lee %A Byungkun Lee %A Jaegul Choo %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-lee23l %I PMLR %P 18988--19009 %U https://proceedings.mlr.press/v202/lee23l.html %V 202 %X Recently, unsupervised representation learning (URL) has improved the sample efficiency of Reinforcement Learning (RL) by pretraining a model from a large unlabeled dataset. The underlying principle of these methods is to learn temporally predictive representations by predicting future states in the latent space. However, an important challenge of this approach is the representational collapse, where the subspace of the latent representations collapses into a low-dimensional manifold. To address this issue, we propose a novel URL framework that causally predicts future states while increasing the dimension of the latent manifold by decorrelating the features in the latent space. Through extensive empirical studies, we demonstrate that our framework effectively learns predictive representations without collapse, which significantly improves the sample efficiency of state-of-the-art URL methods on the Atari 100k benchmark. The code is available at https://github.com/dojeon-ai/SimTPR.
APA
Lee, H., Lee, K., Hwang, D., Lee, H., Lee, B. & Choo, J.. (2023). On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:18988-19009 Available from https://proceedings.mlr.press/v202/lee23l.html.

Related Material