Bias-Variance Tradeoffs for Designing Simultaneous Temporal Experiments

Ruoxuan Xiong, Alex Chin, Sean Taylor
Proceedings of The KDD'23 Workshop on Causal Discovery, Prediction and Decision, PMLR 218:115-131, 2023.

Abstract

We study the analysis and design of simultaneous temporal experiments, where a set of interventions are applied concurrently in continuous time, and outcomes are measured on a sequence of events observed in time. As a motivating setting, suppose multiple data science teams are conducting experiments simultaneously and independently on a ride- hailing platform to test changes to marketplace algorithms such as pricing and matching, and estimating effects from observed event outcomes such as the rate at which ride requests are completed. The design problem involves partitioning a continuous space of time into intervals and assigning treatments at the interval level. Design and analysis must account for three factors: carryover effects from interventions at earlier times, correlation in event outcomes, and effects of interventions tested simultaneously. We provide simulations to build intuition and guidance for practitioners.

Cite this Paper

BibTeX
@InProceedings{pmlr-v218-xiong23a, title = {Bias-Variance Tradeoffs for Designing Simultaneous Temporal Experiments}, author = {Xiong, Ruoxuan and Chin, Alex and Taylor, Sean}, booktitle = {Proceedings of The KDD'23 Workshop on Causal Discovery, Prediction and Decision}, pages = {115--131}, year = {2023}, editor = {Le, Thuc and Li, Jiuyong and Ness, Robert and Triantafillou, Sofia and Shimizu, Shohei and Cui, Peng and Kuang, Kun and Pei, Jian and Wang, Fei and Prosperi, Mattia}, volume = {218}, series = {Proceedings of Machine Learning Research}, month = {07 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v218/xiong23a/xiong23a.pdf}, url = {https://proceedings.mlr.press/v218/xiong23a.html}, abstract = {We study the analysis and design of simultaneous temporal experiments, where a set of interventions are applied concurrently in continuous time, and outcomes are measured on a sequence of events observed in time. As a motivating setting, suppose multiple data science teams are conducting experiments simultaneously and independently on a ride- hailing platform to test changes to marketplace algorithms such as pricing and matching, and estimating effects from observed event outcomes such as the rate at which ride requests are completed. The design problem involves partitioning a continuous space of time into intervals and assigning treatments at the interval level. Design and analysis must account for three factors: carryover effects from interventions at earlier times, correlation in event outcomes, and effects of interventions tested simultaneously. We provide simulations to build intuition and guidance for practitioners.} }
Endnote
%0 Conference Paper %T Bias-Variance Tradeoffs for Designing Simultaneous Temporal Experiments %A Ruoxuan Xiong %A Alex Chin %A Sean Taylor %B Proceedings of The KDD'23 Workshop on Causal Discovery, Prediction and Decision %C Proceedings of Machine Learning Research %D 2023 %E Thuc Le %E Jiuyong Li %E Robert Ness %E Sofia Triantafillou %E Shohei Shimizu %E Peng Cui %E Kun Kuang %E Jian Pei %E Fei Wang %E Mattia Prosperi %F pmlr-v218-xiong23a %I PMLR %P 115--131 %U https://proceedings.mlr.press/v218/xiong23a.html %V 218 %X We study the analysis and design of simultaneous temporal experiments, where a set of interventions are applied concurrently in continuous time, and outcomes are measured on a sequence of events observed in time. As a motivating setting, suppose multiple data science teams are conducting experiments simultaneously and independently on a ride- hailing platform to test changes to marketplace algorithms such as pricing and matching, and estimating effects from observed event outcomes such as the rate at which ride requests are completed. The design problem involves partitioning a continuous space of time into intervals and assigning treatments at the interval level. Design and analysis must account for three factors: carryover effects from interventions at earlier times, correlation in event outcomes, and effects of interventions tested simultaneously. We provide simulations to build intuition and guidance for practitioners.
APA
Xiong, R., Chin, A. & Taylor, S.. (2023). Bias-Variance Tradeoffs for Designing Simultaneous Temporal Experiments. Proceedings of The KDD'23 Workshop on Causal Discovery, Prediction and Decision, in Proceedings of Machine Learning Research 218:115-131 Available from https://proceedings.mlr.press/v218/xiong23a.html.

Related Material