PAC-Bayesian Policy Evaluation for Reinforcement Learning

Mahdi MIlani Fard, Joelle Pineau, Csaba Szepesvari
Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, PMLR R9:233-240, 2011.

Abstract

Bayesian priors offer a compact yet general means of incorporating domain knowledge into many learning tasks. The correctness of the Bayesian analysis and inference, however, largely depends on accuracy and correctness of these priors. PAC-Bayesian methods overcome this problem by providing bounds that hold regardless of the correctness of the prior distribution. This paper introduces the first PAC-Bayesian bound for the batch reinforcement learning problem with function approximation. We show how this bound can be used to perform model-selection in a transfer learning scenario. Our empirical results confirm that PAC-Bayesian policy evaluation is able to leverage prior distributions when they are informative and, unlike standard Bayesian RL approaches, ignore them when they are misleading.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR9-fard11a, title = {{PAC}-{B}ayesian Policy Evaluation for Reinforcement Learning}, author = {Fard, Mahdi MIlani and Pineau, Joelle and Szepesvari, Csaba}, booktitle = {Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence}, pages = {233--240}, year = {2011}, editor = {Cozman, Fabio and Pfeffer, Avi}, volume = {R9}, series = {Proceedings of Machine Learning Research}, month = {14--17 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r9/main/assets/fard11a/fard11a.pdf}, url = {https://proceedings.mlr.press/r9/fard11a.html}, abstract = {Bayesian priors offer a compact yet general means of incorporating domain knowledge into many learning tasks. The correctness of the Bayesian analysis and inference, however, largely depends on accuracy and correctness of these priors. PAC-Bayesian methods overcome this problem by providing bounds that hold regardless of the correctness of the prior distribution. This paper introduces the first PAC-Bayesian bound for the batch reinforcement learning problem with function approximation. We show how this bound can be used to perform model-selection in a transfer learning scenario. Our empirical results confirm that PAC-Bayesian policy evaluation is able to leverage prior distributions when they are informative and, unlike standard Bayesian RL approaches, ignore them when they are misleading.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T PAC-Bayesian Policy Evaluation for Reinforcement Learning %A Mahdi MIlani Fard %A Joelle Pineau %A Csaba Szepesvari %B Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2011 %E Fabio Cozman %E Avi Pfeffer %F pmlr-vR9-fard11a %I PMLR %P 233--240 %U https://proceedings.mlr.press/r9/fard11a.html %V R9 %X Bayesian priors offer a compact yet general means of incorporating domain knowledge into many learning tasks. The correctness of the Bayesian analysis and inference, however, largely depends on accuracy and correctness of these priors. PAC-Bayesian methods overcome this problem by providing bounds that hold regardless of the correctness of the prior distribution. This paper introduces the first PAC-Bayesian bound for the batch reinforcement learning problem with function approximation. We show how this bound can be used to perform model-selection in a transfer learning scenario. Our empirical results confirm that PAC-Bayesian policy evaluation is able to leverage prior distributions when they are informative and, unlike standard Bayesian RL approaches, ignore them when they are misleading. %Z Reissued by PMLR on 04 October 2026.
APA
Fard, M.M., Pineau, J. & Szepesvari, C.. (2011). PAC-Bayesian Policy Evaluation for Reinforcement Learning. Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R9:233-240 Available from https://proceedings.mlr.press/r9/fard11a.html. Reissued by PMLR on 04 October 2026.

Related Material