Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits

Miroslav Dudik, Dumitru Erhan, John Langford, Lihong Li
Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, PMLR R10:245-252, 2012.

Abstract

We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In addition, our approach allows generating longer histories by careful control of a bias-variance tradeoff, and further decreases variance by incorporating information about randomness of the target policy. Empirical evidence from synthetic and realworld exploration learning problems shows the new evaluator successfully unifies previous approaches and uses information an order of magnitude more efficiently.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR10-dudik12a, title = {Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits}, author = {Dudik, Miroslav and Erhan, Dumitru and Langford, John and Li, Lihong}, booktitle = {Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence}, pages = {245--252}, year = {2012}, editor = {de Freitas, Nando and Murphy, Kevin}, volume = {R10}, series = {Proceedings of Machine Learning Research}, month = {14--18 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r10/main/assets/dudik12a/dudik12a.pdf}, url = {https://proceedings.mlr.press/r10/dudik12a.html}, abstract = {We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In addition, our approach allows generating longer histories by careful control of a bias-variance tradeoff, and further decreases variance by incorporating information about randomness of the target policy. Empirical evidence from synthetic and realworld exploration learning problems shows the new evaluator successfully unifies previous approaches and uses information an order of magnitude more efficiently.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits %A Miroslav Dudik %A Dumitru Erhan %A John Langford %A Lihong Li %B Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2012 %E Nando de Freitas %E Kevin Murphy %F pmlr-vR10-dudik12a %I PMLR %P 245--252 %U https://proceedings.mlr.press/r10/dudik12a.html %V R10 %X We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In addition, our approach allows generating longer histories by careful control of a bias-variance tradeoff, and further decreases variance by incorporating information about randomness of the target policy. Empirical evidence from synthetic and realworld exploration learning problems shows the new evaluator successfully unifies previous approaches and uses information an order of magnitude more efficiently. %Z Reissued by PMLR on 04 October 2026.
APA
Dudik, M., Erhan, D., Langford, J. & Li, L.. (2012). Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits. Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R10:245-252 Available from https://proceedings.mlr.press/r10/dudik12a.html. Reissued by PMLR on 04 October 2026.

Related Material