Uniformly Conservative Exploration in Reinforcement Learning

Wanqiao Xu, Yecheng Ma, Kan Xu, Hamsa Bastani, Osbert Bastani
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:10856-10870, 2023.

Abstract

A key challenge to deploying reinforcement learning in practice is avoiding excessive (harmful) exploration in individual episodes. We propose a natural constraint on exploration—uniformly outperforming a conservative policy (adaptively estimated from all data observed thus far), up to a per-episode exploration budget. We design a novel algorithm that uses a UCB reinforcement learning policy for exploration, but overrides it as needed to satisfy our exploration constraint with high probability. Importantly, to ensure unbiased exploration across the state space, our algorithm adaptively determines when to explore. We prove that our approach remains conservative while minimizing regret in the tabular setting. We experimentally validate our results on a sepsis treatment task and an HIV treatment task, demonstrating that our algorithm can learn while ensuring good performance compared to the baseline policy for every patient; the latter task also demonstrates that our approach extends to continuous state spaces via deep reinforcement learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-xu23j, title = {Uniformly Conservative Exploration in Reinforcement Learning}, author = {Xu, Wanqiao and Ma, Yecheng and Xu, Kan and Bastani, Hamsa and Bastani, Osbert}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {10856--10870}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/xu23j/xu23j.pdf}, url = {https://proceedings.mlr.press/v206/xu23j.html}, abstract = {A key challenge to deploying reinforcement learning in practice is avoiding excessive (harmful) exploration in individual episodes. We propose a natural constraint on exploration—uniformly outperforming a conservative policy (adaptively estimated from all data observed thus far), up to a per-episode exploration budget. We design a novel algorithm that uses a UCB reinforcement learning policy for exploration, but overrides it as needed to satisfy our exploration constraint with high probability. Importantly, to ensure unbiased exploration across the state space, our algorithm adaptively determines when to explore. We prove that our approach remains conservative while minimizing regret in the tabular setting. We experimentally validate our results on a sepsis treatment task and an HIV treatment task, demonstrating that our algorithm can learn while ensuring good performance compared to the baseline policy for every patient; the latter task also demonstrates that our approach extends to continuous state spaces via deep reinforcement learning.} }
Endnote
%0 Conference Paper %T Uniformly Conservative Exploration in Reinforcement Learning %A Wanqiao Xu %A Yecheng Ma %A Kan Xu %A Hamsa Bastani %A Osbert Bastani %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-xu23j %I PMLR %P 10856--10870 %U https://proceedings.mlr.press/v206/xu23j.html %V 206 %X A key challenge to deploying reinforcement learning in practice is avoiding excessive (harmful) exploration in individual episodes. We propose a natural constraint on exploration—uniformly outperforming a conservative policy (adaptively estimated from all data observed thus far), up to a per-episode exploration budget. We design a novel algorithm that uses a UCB reinforcement learning policy for exploration, but overrides it as needed to satisfy our exploration constraint with high probability. Importantly, to ensure unbiased exploration across the state space, our algorithm adaptively determines when to explore. We prove that our approach remains conservative while minimizing regret in the tabular setting. We experimentally validate our results on a sepsis treatment task and an HIV treatment task, demonstrating that our algorithm can learn while ensuring good performance compared to the baseline policy for every patient; the latter task also demonstrates that our approach extends to continuous state spaces via deep reinforcement learning.
APA
Xu, W., Ma, Y., Xu, K., Bastani, H. & Bastani, O.. (2023). Uniformly Conservative Exploration in Reinforcement Learning. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:10856-10870 Available from https://proceedings.mlr.press/v206/xu23j.html.

Related Material