Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments

Qianglin Wen, Chengchun Shi, Ying Yang, Niansheng Tang, Hongtu Zhu
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:66481-66519, 2025.

Abstract

A/B testing has become the gold standard for modern technological industries for policy evaluation. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the autocorrelations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to all the aforementioned estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-wen25d, title = {Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments}, author = {Wen, Qianglin and Shi, Chengchun and Yang, Ying and Tang, Niansheng and Zhu, Hongtu}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {66481--66519}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/wen25d/wen25d.pdf}, url = {https://proceedings.mlr.press/v267/wen25d.html}, abstract = {A/B testing has become the gold standard for modern technological industries for policy evaluation. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the autocorrelations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to all the aforementioned estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.} }
Endnote
%0 Conference Paper %T Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments %A Qianglin Wen %A Chengchun Shi %A Ying Yang %A Niansheng Tang %A Hongtu Zhu %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-wen25d %I PMLR %P 66481--66519 %U https://proceedings.mlr.press/v267/wen25d.html %V 267 %X A/B testing has become the gold standard for modern technological industries for policy evaluation. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the autocorrelations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to all the aforementioned estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.
APA
Wen, Q., Shi, C., Yang, Y., Tang, N. & Zhu, H.. (2025). Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:66481-66519 Available from https://proceedings.mlr.press/v267/wen25d.html.

Related Material