Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations

Stefan Sylvius Wagner, Stefan Harmeling
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:49788-49807, 2024.

Abstract

In this paper we adopt a representation-centric perspective on exploration in reinforcement learning, viewing exploration fundamentally as a density estimation problem. We investigate the effectiveness of clustering representations for exploration in 3-D environments, based on the observation that the importance of pixel changes between transitions is less pronounced in 3-D environments compared to 2-D environments, where pixel changes between transitions are typically distinct and significant. We propose a method that performs episodic and global clustering on random representations and on pre-trained DINO representations to count states, i.e, estimate pseudo-counts. Surprisingly, even random features can be clustered effectively to count states in 3-D environments, however when these become visually more complex, pre-trained DINO representations are more effective thanks to the pre-trained inductive biases in the representations. Overall, this presents a pathway for integrating pre-trained biases into exploration. We evaluate our approach on the VizDoom and Habitat environments, demonstrating that our method surpasses other well-known exploration methods in these settings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-wagner24a, title = {Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations}, author = {Wagner, Stefan Sylvius and Harmeling, Stefan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {49788--49807}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/wagner24a/wagner24a.pdf}, url = {https://proceedings.mlr.press/v235/wagner24a.html}, abstract = {In this paper we adopt a representation-centric perspective on exploration in reinforcement learning, viewing exploration fundamentally as a density estimation problem. We investigate the effectiveness of clustering representations for exploration in 3-D environments, based on the observation that the importance of pixel changes between transitions is less pronounced in 3-D environments compared to 2-D environments, where pixel changes between transitions are typically distinct and significant. We propose a method that performs episodic and global clustering on random representations and on pre-trained DINO representations to count states, i.e, estimate pseudo-counts. Surprisingly, even random features can be clustered effectively to count states in 3-D environments, however when these become visually more complex, pre-trained DINO representations are more effective thanks to the pre-trained inductive biases in the representations. Overall, this presents a pathway for integrating pre-trained biases into exploration. We evaluate our approach on the VizDoom and Habitat environments, demonstrating that our method surpasses other well-known exploration methods in these settings.} }
Endnote
%0 Conference Paper %T Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations %A Stefan Sylvius Wagner %A Stefan Harmeling %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-wagner24a %I PMLR %P 49788--49807 %U https://proceedings.mlr.press/v235/wagner24a.html %V 235 %X In this paper we adopt a representation-centric perspective on exploration in reinforcement learning, viewing exploration fundamentally as a density estimation problem. We investigate the effectiveness of clustering representations for exploration in 3-D environments, based on the observation that the importance of pixel changes between transitions is less pronounced in 3-D environments compared to 2-D environments, where pixel changes between transitions are typically distinct and significant. We propose a method that performs episodic and global clustering on random representations and on pre-trained DINO representations to count states, i.e, estimate pseudo-counts. Surprisingly, even random features can be clustered effectively to count states in 3-D environments, however when these become visually more complex, pre-trained DINO representations are more effective thanks to the pre-trained inductive biases in the representations. Overall, this presents a pathway for integrating pre-trained biases into exploration. We evaluate our approach on the VizDoom and Habitat environments, demonstrating that our method surpasses other well-known exploration methods in these settings.
APA
Wagner, S.S. & Harmeling, S.. (2024). Just Cluster It: An Approach for Exploration in High-Dimensions using Clustering and Pre-Trained Representations. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:49788-49807 Available from https://proceedings.mlr.press/v235/wagner24a.html.

Related Material