Conservative Inference in Switchback Experiments

Jose Blanchet, Peter Glynn, Ramesh Johari, Linjia Wu, Wenqian Xing
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2422-2430, 2026.

Abstract

Switchback experiments are widely used in dynamic systems—such as ridesharing platforms and online marketplaces—to evaluate interventions under interference. However, the standard pipeline of estimating the average treatment effect (ATE) with a difference-in-means (DM) estimator can exhibit systematic bias in dynamic settings with evolving system state, due to intertemporal dependence (“carryover effects"). In this paper, we study this bias in a continuous-time Markov chain model of switchback experiments with stochastically monotone dynamics and state-monotone rewards; these are reasonable representations of mean-reverting and auto-regressive systems. We show the DM estimator systematically underestimates the true ATE, because it targets an average of transient treatment effects rather than the ATE itself. Using the Ornstein-Uhlenbeck process as a tractable example, we derive closed-form expressions for bias and variance; this analysis shows that standard approaches overestimate the true variance. Taken together, these effects mean that standard switchback experiment analysis yields overly conservative inference. We validate our theory using a ride-sharing simulation with real-world calibration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-blanchet26a, title = { Conservative Inference in Switchback Experiments }, author = {Blanchet, Jose and Glynn, Peter and Johari, Ramesh and Wu, Linjia and Xing, Wenqian}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2422--2430}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/blanchet26a/blanchet26a.pdf}, url = {https://proceedings.mlr.press/v300/blanchet26a.html}, abstract = { Switchback experiments are widely used in dynamic systems—such as ridesharing platforms and online marketplaces—to evaluate interventions under interference. However, the standard pipeline of estimating the average treatment effect (ATE) with a difference-in-means (DM) estimator can exhibit systematic bias in dynamic settings with evolving system state, due to intertemporal dependence (“carryover effects"). In this paper, we study this bias in a continuous-time Markov chain model of switchback experiments with stochastically monotone dynamics and state-monotone rewards; these are reasonable representations of mean-reverting and auto-regressive systems. We show the DM estimator systematically underestimates the true ATE, because it targets an average of transient treatment effects rather than the ATE itself. Using the Ornstein-Uhlenbeck process as a tractable example, we derive closed-form expressions for bias and variance; this analysis shows that standard approaches overestimate the true variance. Taken together, these effects mean that standard switchback experiment analysis yields overly conservative inference. We validate our theory using a ride-sharing simulation with real-world calibration. } }
Endnote
%0 Conference Paper %T Conservative Inference in Switchback Experiments %A Jose Blanchet %A Peter Glynn %A Ramesh Johari %A Linjia Wu %A Wenqian Xing %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-blanchet26a %I PMLR %P 2422--2430 %U https://proceedings.mlr.press/v300/blanchet26a.html %V 300 %X Switchback experiments are widely used in dynamic systems—such as ridesharing platforms and online marketplaces—to evaluate interventions under interference. However, the standard pipeline of estimating the average treatment effect (ATE) with a difference-in-means (DM) estimator can exhibit systematic bias in dynamic settings with evolving system state, due to intertemporal dependence (“carryover effects"). In this paper, we study this bias in a continuous-time Markov chain model of switchback experiments with stochastically monotone dynamics and state-monotone rewards; these are reasonable representations of mean-reverting and auto-regressive systems. We show the DM estimator systematically underestimates the true ATE, because it targets an average of transient treatment effects rather than the ATE itself. Using the Ornstein-Uhlenbeck process as a tractable example, we derive closed-form expressions for bias and variance; this analysis shows that standard approaches overestimate the true variance. Taken together, these effects mean that standard switchback experiment analysis yields overly conservative inference. We validate our theory using a ride-sharing simulation with real-world calibration.
APA
Blanchet, J., Glynn, P., Johari, R., Wu, L. & Xing, W.. (2026). Conservative Inference in Switchback Experiments . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2422-2430 Available from https://proceedings.mlr.press/v300/blanchet26a.html.

Related Material