RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation

Zelei Cheng, Xian Wu, Jiahao Yu, Sabrina Yang, Gang Wang, Xinyu Xing
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:8203-8228, 2024.

Abstract

Deep reinforcement learning (DRL) is playing an increasingly important role in real-world applications. However, obtaining an optimally performing DRL agent for complex tasks, especially with sparse rewards, remains a significant challenge. The training of a DRL agent can be often trapped in a bottleneck without further progress. In this paper, we propose RICE, an innovative refining scheme for reinforcement learning that incorporates explanation methods to break through the training bottlenecks. The high-level idea of RICE is to construct a new initial state distribution that combines both the default initial states and critical states identified through explanation methods, thereby encouraging the agent to explore from the mixed initial states. Through careful design, we can theoretically guarantee that our refining scheme has a tighter sub-optimality bound. We evaluate RICE in various popular RL environments and real-world applications. The results demonstrate that RICE significantly outperforms existing refining schemes in enhancing agent performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-cheng24j, title = {{RICE}: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation}, author = {Cheng, Zelei and Wu, Xian and Yu, Jiahao and Yang, Sabrina and Wang, Gang and Xing, Xinyu}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {8203--8228}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/cheng24j/cheng24j.pdf}, url = {https://proceedings.mlr.press/v235/cheng24j.html}, abstract = {Deep reinforcement learning (DRL) is playing an increasingly important role in real-world applications. However, obtaining an optimally performing DRL agent for complex tasks, especially with sparse rewards, remains a significant challenge. The training of a DRL agent can be often trapped in a bottleneck without further progress. In this paper, we propose RICE, an innovative refining scheme for reinforcement learning that incorporates explanation methods to break through the training bottlenecks. The high-level idea of RICE is to construct a new initial state distribution that combines both the default initial states and critical states identified through explanation methods, thereby encouraging the agent to explore from the mixed initial states. Through careful design, we can theoretically guarantee that our refining scheme has a tighter sub-optimality bound. We evaluate RICE in various popular RL environments and real-world applications. The results demonstrate that RICE significantly outperforms existing refining schemes in enhancing agent performance.} }
Endnote
%0 Conference Paper %T RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation %A Zelei Cheng %A Xian Wu %A Jiahao Yu %A Sabrina Yang %A Gang Wang %A Xinyu Xing %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-cheng24j %I PMLR %P 8203--8228 %U https://proceedings.mlr.press/v235/cheng24j.html %V 235 %X Deep reinforcement learning (DRL) is playing an increasingly important role in real-world applications. However, obtaining an optimally performing DRL agent for complex tasks, especially with sparse rewards, remains a significant challenge. The training of a DRL agent can be often trapped in a bottleneck without further progress. In this paper, we propose RICE, an innovative refining scheme for reinforcement learning that incorporates explanation methods to break through the training bottlenecks. The high-level idea of RICE is to construct a new initial state distribution that combines both the default initial states and critical states identified through explanation methods, thereby encouraging the agent to explore from the mixed initial states. Through careful design, we can theoretically guarantee that our refining scheme has a tighter sub-optimality bound. We evaluate RICE in various popular RL environments and real-world applications. The results demonstrate that RICE significantly outperforms existing refining schemes in enhancing agent performance.
APA
Cheng, Z., Wu, X., Yu, J., Yang, S., Wang, G. & Xing, X.. (2024). RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:8203-8228 Available from https://proceedings.mlr.press/v235/cheng24j.html.

Related Material