Task-specific experimental design for treatment effect estimation

Bethany Connolly, Kim Moore, Tobias Schwedes, Alexander Adam, Gary Willis, Ilya Feige, Christopher Frye
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:6384-6401, 2023.

Abstract

Understanding causality should be a core requirement of any attempt to build real impact through AI. Due to the inherent unobservability of counterfactuals, large randomised trials (RCTs) are the standard for causal inference. But large experiments are generically expensive, and randomisation carries its own costs, e.g. when suboptimal decisions are trialed. Recent work has proposed more sample-efficient alternatives to RCTs, but these are not adaptable to the downstream application for which the causal effect is sought. In this work, we develop a task-specific approach to experimental design and derive sampling strategies customised to particular downstream applications. Across a range of important tasks, real-world datasets, and sample sizes, our method outperforms other benchmarks, e.g. requiring an order-of-magnitude less data to match RCT performance on targeted marketing tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-connolly23a, title = {Task-specific experimental design for treatment effect estimation}, author = {Connolly, Bethany and Moore, Kim and Schwedes, Tobias and Adam, Alexander and Willis, Gary and Feige, Ilya and Frye, Christopher}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {6384--6401}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/connolly23a/connolly23a.pdf}, url = {https://proceedings.mlr.press/v202/connolly23a.html}, abstract = {Understanding causality should be a core requirement of any attempt to build real impact through AI. Due to the inherent unobservability of counterfactuals, large randomised trials (RCTs) are the standard for causal inference. But large experiments are generically expensive, and randomisation carries its own costs, e.g. when suboptimal decisions are trialed. Recent work has proposed more sample-efficient alternatives to RCTs, but these are not adaptable to the downstream application for which the causal effect is sought. In this work, we develop a task-specific approach to experimental design and derive sampling strategies customised to particular downstream applications. Across a range of important tasks, real-world datasets, and sample sizes, our method outperforms other benchmarks, e.g. requiring an order-of-magnitude less data to match RCT performance on targeted marketing tasks.} }
Endnote
%0 Conference Paper %T Task-specific experimental design for treatment effect estimation %A Bethany Connolly %A Kim Moore %A Tobias Schwedes %A Alexander Adam %A Gary Willis %A Ilya Feige %A Christopher Frye %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-connolly23a %I PMLR %P 6384--6401 %U https://proceedings.mlr.press/v202/connolly23a.html %V 202 %X Understanding causality should be a core requirement of any attempt to build real impact through AI. Due to the inherent unobservability of counterfactuals, large randomised trials (RCTs) are the standard for causal inference. But large experiments are generically expensive, and randomisation carries its own costs, e.g. when suboptimal decisions are trialed. Recent work has proposed more sample-efficient alternatives to RCTs, but these are not adaptable to the downstream application for which the causal effect is sought. In this work, we develop a task-specific approach to experimental design and derive sampling strategies customised to particular downstream applications. Across a range of important tasks, real-world datasets, and sample sizes, our method outperforms other benchmarks, e.g. requiring an order-of-magnitude less data to match RCT performance on targeted marketing tasks.
APA
Connolly, B., Moore, K., Schwedes, T., Adam, A., Willis, G., Feige, I. & Frye, C.. (2023). Task-specific experimental design for treatment effect estimation. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:6384-6401 Available from https://proceedings.mlr.press/v202/connolly23a.html.

Related Material