Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

Sam Lobel, Akhil Bagaria, George Konidaris
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:22594-22613, 2023.

Abstract

We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state’s visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma’s Revenge.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-lobel23a, title = {Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning}, author = {Lobel, Sam and Bagaria, Akhil and Konidaris, George}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {22594--22613}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/lobel23a/lobel23a.pdf}, url = {https://proceedings.mlr.press/v202/lobel23a.html}, abstract = {We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state’s visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma’s Revenge.} }
Endnote
%0 Conference Paper %T Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning %A Sam Lobel %A Akhil Bagaria %A George Konidaris %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-lobel23a %I PMLR %P 22594--22613 %U https://proceedings.mlr.press/v202/lobel23a.html %V 202 %X We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state’s visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma’s Revenge.
APA
Lobel, S., Bagaria, A. & Konidaris, G.. (2023). Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:22594-22613 Available from https://proceedings.mlr.press/v202/lobel23a.html.

Related Material