On Online Experimentation without Device Identifiers

Shiv Shankar, Ritwik Sinha, Madalina Fiterau
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:44394-44412, 2024.

Abstract

Measuring human feedback via randomized experimentation is a cornerstone of data-driven decision-making. The methodology used to estimate user preferences from their online behaviours is critically dependent on user identifiers. However, in today’s digital landscape, consumers frequently interact with content across multiple devices, which are often recorded with different identifiers for the same consumer. The inability to match different device identities across consumers poses significant challenges for accurately estimating human preferences and other causal effects. Moreover, without strong assumptions about the device-user graph, the causal effects might not be identifiable. In this paper, we propose HIFIVE, a variational method to solve the problem of estimating global average treatment effects (GATE) from a fragmented view of exposures and outcomes. Experiments show that our estimator is superior to standard estimators, with a lower bias and greater robustness to network uncertainty.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-shankar24a, title = {On Online Experimentation without Device Identifiers}, author = {Shankar, Shiv and Sinha, Ritwik and Fiterau, Madalina}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {44394--44412}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/shankar24a/shankar24a.pdf}, url = {https://proceedings.mlr.press/v235/shankar24a.html}, abstract = {Measuring human feedback via randomized experimentation is a cornerstone of data-driven decision-making. The methodology used to estimate user preferences from their online behaviours is critically dependent on user identifiers. However, in today’s digital landscape, consumers frequently interact with content across multiple devices, which are often recorded with different identifiers for the same consumer. The inability to match different device identities across consumers poses significant challenges for accurately estimating human preferences and other causal effects. Moreover, without strong assumptions about the device-user graph, the causal effects might not be identifiable. In this paper, we propose HIFIVE, a variational method to solve the problem of estimating global average treatment effects (GATE) from a fragmented view of exposures and outcomes. Experiments show that our estimator is superior to standard estimators, with a lower bias and greater robustness to network uncertainty.} }
Endnote
%0 Conference Paper %T On Online Experimentation without Device Identifiers %A Shiv Shankar %A Ritwik Sinha %A Madalina Fiterau %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-shankar24a %I PMLR %P 44394--44412 %U https://proceedings.mlr.press/v235/shankar24a.html %V 235 %X Measuring human feedback via randomized experimentation is a cornerstone of data-driven decision-making. The methodology used to estimate user preferences from their online behaviours is critically dependent on user identifiers. However, in today’s digital landscape, consumers frequently interact with content across multiple devices, which are often recorded with different identifiers for the same consumer. The inability to match different device identities across consumers poses significant challenges for accurately estimating human preferences and other causal effects. Moreover, without strong assumptions about the device-user graph, the causal effects might not be identifiable. In this paper, we propose HIFIVE, a variational method to solve the problem of estimating global average treatment effects (GATE) from a fragmented view of exposures and outcomes. Experiments show that our estimator is superior to standard estimators, with a lower bias and greater robustness to network uncertainty.
APA
Shankar, S., Sinha, R. & Fiterau, M.. (2024). On Online Experimentation without Device Identifiers. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:44394-44412 Available from https://proceedings.mlr.press/v235/shankar24a.html.

Related Material