Sequential Off-Policy Learning with Logarithmic Smoothing

Maxime Haddouche, Otmane Sakhi
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:1846-1854, 2026.

Abstract

Off-policy learning enables training policies from logged interaction data. Most prior work considers the batch setting, where a policy is learned from data generated by a single behavior policy. In real systems, however, policies are updated and redeployed repeatedly, each time training on all previously collected data while generating new interactions for future updates. This sequential off-policy learning setting is common in practice but remains largely unexplored theoretically. In this work, we present and study a simple algorithm for \emph{sequential off-policy learning}, combining Logarithmic Smoothing (LS) estimation with online PAC-Bayesian tools. We further show that a principled adjustment to LS improves performance and accelerates convergence under mild conditions. The algorithms introduced generalize previous work: they match state-of-the-art offline approaches in the batch case and substantially outperform them when policies are updated sequentially. Empirical evaluations highlight both the benefits of the sequential framework and the strength of the proposed algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-haddouche26a, title = { Sequential Off-Policy Learning with Logarithmic Smoothing }, author = {Haddouche, Maxime and Sakhi, Otmane}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {1846--1854}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/haddouche26a/haddouche26a.pdf}, url = {https://proceedings.mlr.press/v300/haddouche26a.html}, abstract = { Off-policy learning enables training policies from logged interaction data. Most prior work considers the batch setting, where a policy is learned from data generated by a single behavior policy. In real systems, however, policies are updated and redeployed repeatedly, each time training on all previously collected data while generating new interactions for future updates. This sequential off-policy learning setting is common in practice but remains largely unexplored theoretically. In this work, we present and study a simple algorithm for \emph{sequential off-policy learning}, combining Logarithmic Smoothing (LS) estimation with online PAC-Bayesian tools. We further show that a principled adjustment to LS improves performance and accelerates convergence under mild conditions. The algorithms introduced generalize previous work: they match state-of-the-art offline approaches in the batch case and substantially outperform them when policies are updated sequentially. Empirical evaluations highlight both the benefits of the sequential framework and the strength of the proposed algorithms. } }
Endnote
%0 Conference Paper %T Sequential Off-Policy Learning with Logarithmic Smoothing %A Maxime Haddouche %A Otmane Sakhi %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-haddouche26a %I PMLR %P 1846--1854 %U https://proceedings.mlr.press/v300/haddouche26a.html %V 300 %X Off-policy learning enables training policies from logged interaction data. Most prior work considers the batch setting, where a policy is learned from data generated by a single behavior policy. In real systems, however, policies are updated and redeployed repeatedly, each time training on all previously collected data while generating new interactions for future updates. This sequential off-policy learning setting is common in practice but remains largely unexplored theoretically. In this work, we present and study a simple algorithm for \emph{sequential off-policy learning}, combining Logarithmic Smoothing (LS) estimation with online PAC-Bayesian tools. We further show that a principled adjustment to LS improves performance and accelerates convergence under mild conditions. The algorithms introduced generalize previous work: they match state-of-the-art offline approaches in the batch case and substantially outperform them when policies are updated sequentially. Empirical evaluations highlight both the benefits of the sequential framework and the strength of the proposed algorithms.
APA
Haddouche, M. & Sakhi, O.. (2026). Sequential Off-Policy Learning with Logarithmic Smoothing . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:1846-1854 Available from https://proceedings.mlr.press/v300/haddouche26a.html.

Related Material