Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling

Imad Aouali, Victor-Emmanuel Brunel, David Rohde, Anna Korba
Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, PMLR 244:88-109, 2024.

Abstract

Off-policy learning (OPL) often involves minimizing a risk estimator based on importance weighting to correct bias from the logging policy used to collect data. However, this method can produce an estimator with a high variance. A common solution is to regularize the importance weights and learn the policy by minimizing an estimator with penalties derived from generalization bounds specific to the estimator. This approach, known as pessimism, has gained recent attention but lacks a unified framework for analysis. To address this gap, we introduce a comprehensive PAC-Bayesian framework to examine pessimism with regularized importance weighting. We derive a tractable PAC-Bayesian generalization bound that universally applies to common importance weight regularizations, enabling their comparison within a single framework. Our empirical results challenge common understanding, demonstrating the effectiveness of standard IW regularization techniques.

Cite this Paper

BibTeX
@InProceedings{pmlr-v244-aouali24a, title = {Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling}, author = {Aouali, Imad and Brunel, Victor-Emmanuel and Rohde, David and Korba, Anna}, booktitle = {Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence}, pages = {88--109}, year = {2024}, editor = {Kiyavash, Negar and Mooij, Joris M.}, volume = {244}, series = {Proceedings of Machine Learning Research}, month = {15--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v244/main/assets/aouali24a/aouali24a.pdf}, url = {https://proceedings.mlr.press/v244/aouali24a.html}, abstract = {Off-policy learning (OPL) often involves minimizing a risk estimator based on importance weighting to correct bias from the logging policy used to collect data. However, this method can produce an estimator with a high variance. A common solution is to regularize the importance weights and learn the policy by minimizing an estimator with penalties derived from generalization bounds specific to the estimator. This approach, known as pessimism, has gained recent attention but lacks a unified framework for analysis. To address this gap, we introduce a comprehensive PAC-Bayesian framework to examine pessimism with regularized importance weighting. We derive a tractable PAC-Bayesian generalization bound that universally applies to common importance weight regularizations, enabling their comparison within a single framework. Our empirical results challenge common understanding, demonstrating the effectiveness of standard IW regularization techniques.} }
Endnote
%0 Conference Paper %T Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling %A Imad Aouali %A Victor-Emmanuel Brunel %A David Rohde %A Anna Korba %B Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2024 %E Negar Kiyavash %E Joris M. Mooij %F pmlr-v244-aouali24a %I PMLR %P 88--109 %U https://proceedings.mlr.press/v244/aouali24a.html %V 244 %X Off-policy learning (OPL) often involves minimizing a risk estimator based on importance weighting to correct bias from the logging policy used to collect data. However, this method can produce an estimator with a high variance. A common solution is to regularize the importance weights and learn the policy by minimizing an estimator with penalties derived from generalization bounds specific to the estimator. This approach, known as pessimism, has gained recent attention but lacks a unified framework for analysis. To address this gap, we introduce a comprehensive PAC-Bayesian framework to examine pessimism with regularized importance weighting. We derive a tractable PAC-Bayesian generalization bound that universally applies to common importance weight regularizations, enabling their comparison within a single framework. Our empirical results challenge common understanding, demonstrating the effectiveness of standard IW regularization techniques.
APA
Aouali, I., Brunel, V., Rohde, D. & Korba, A.. (2024). Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling. Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 244:88-109 Available from https://proceedings.mlr.press/v244/aouali24a.html.

Related Material