Random Latent Exploration for Deep Reinforcement Learning

Srinath V. Mahankali, Zhang-Wei Hong, Ayush Sekhari, Alexander Rakhlin, Pulkit Agrawal
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:34219-34252, 2024.

Abstract

The ability to efficiently explore high-dimensional state spaces is essential for the practical success of deep Reinforcement Learning (RL). This paper introduces a new exploration technique called Random Latent Exploration (RLE), that combines the strengths of exploration bonuses and randomized value functions (two popular approaches for effective exploration in deep RL). RLE leverages the idea of perturbing rewards by adding structured random rewards to the original task rewards in certain (random) states of the environment, to encourage the agent to explore the environment during training. RLE is straightforward to implement and performs well in practice. To demonstrate the practical effectiveness of RLE, we evaluate it on the challenging Atari and IsaacGym benchmarks and show that RLE exhibits higher overall scores across all the tasks than other approaches, including action-noise and randomized value function exploration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-mahankali24a, title = {Random Latent Exploration for Deep Reinforcement Learning}, author = {Mahankali, Srinath V. and Hong, Zhang-Wei and Sekhari, Ayush and Rakhlin, Alexander and Agrawal, Pulkit}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {34219--34252}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/mahankali24a/mahankali24a.pdf}, url = {https://proceedings.mlr.press/v235/mahankali24a.html}, abstract = {The ability to efficiently explore high-dimensional state spaces is essential for the practical success of deep Reinforcement Learning (RL). This paper introduces a new exploration technique called Random Latent Exploration (RLE), that combines the strengths of exploration bonuses and randomized value functions (two popular approaches for effective exploration in deep RL). RLE leverages the idea of perturbing rewards by adding structured random rewards to the original task rewards in certain (random) states of the environment, to encourage the agent to explore the environment during training. RLE is straightforward to implement and performs well in practice. To demonstrate the practical effectiveness of RLE, we evaluate it on the challenging Atari and IsaacGym benchmarks and show that RLE exhibits higher overall scores across all the tasks than other approaches, including action-noise and randomized value function exploration.} }
Endnote
%0 Conference Paper %T Random Latent Exploration for Deep Reinforcement Learning %A Srinath V. Mahankali %A Zhang-Wei Hong %A Ayush Sekhari %A Alexander Rakhlin %A Pulkit Agrawal %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-mahankali24a %I PMLR %P 34219--34252 %U https://proceedings.mlr.press/v235/mahankali24a.html %V 235 %X The ability to efficiently explore high-dimensional state spaces is essential for the practical success of deep Reinforcement Learning (RL). This paper introduces a new exploration technique called Random Latent Exploration (RLE), that combines the strengths of exploration bonuses and randomized value functions (two popular approaches for effective exploration in deep RL). RLE leverages the idea of perturbing rewards by adding structured random rewards to the original task rewards in certain (random) states of the environment, to encourage the agent to explore the environment during training. RLE is straightforward to implement and performs well in practice. To demonstrate the practical effectiveness of RLE, we evaluate it on the challenging Atari and IsaacGym benchmarks and show that RLE exhibits higher overall scores across all the tasks than other approaches, including action-noise and randomized value function exploration.
APA
Mahankali, S.V., Hong, Z., Sekhari, A., Rakhlin, A. & Agrawal, P.. (2024). Random Latent Exploration for Deep Reinforcement Learning. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:34219-34252 Available from https://proceedings.mlr.press/v235/mahankali24a.html.

Related Material