Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments

Guillaume W. Basse, Hossein Azari Soufiani, Diane Lambert
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR 51:1412-1420, 2016.

Abstract

Buyers (e.g., advertisers) often have limited financial or processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling, and any change may affect what buyers pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is an unbiased estimator when all the bidders in the same auction are randomly assigned to A or B but the observed treatment effect can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be much reduced if separate budgets are maintained for the A and B arms of the experiment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v51-basse16b, title = {Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments}, author = {Basse, Guillaume W. and Azari Soufiani, Hossein and Lambert, Diane}, booktitle = {Proceedings of the 19th International Conference on Artificial Intelligence and Statistics}, pages = {1412--1420}, year = {2016}, editor = {Gretton, Arthur and Robert, Christian C.}, volume = {51}, series = {Proceedings of Machine Learning Research}, address = {Cadiz, Spain}, month = {09--11 May}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v51/basse16b.pdf}, url = {https://proceedings.mlr.press/v51/basse16b.html}, abstract = {Buyers (e.g., advertisers) often have limited financial or processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling, and any change may affect what buyers pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is an unbiased estimator when all the bidders in the same auction are randomly assigned to A or B but the observed treatment effect can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be much reduced if separate budgets are maintained for the A and B arms of the experiment.} }
Endnote
%0 Conference Paper %T Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments %A Guillaume W. Basse %A Hossein Azari Soufiani %A Diane Lambert %B Proceedings of the 19th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2016 %E Arthur Gretton %E Christian C. Robert %F pmlr-v51-basse16b %I PMLR %P 1412--1420 %U https://proceedings.mlr.press/v51/basse16b.html %V 51 %X Buyers (e.g., advertisers) often have limited financial or processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling, and any change may affect what buyers pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is an unbiased estimator when all the bidders in the same auction are randomly assigned to A or B but the observed treatment effect can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be much reduced if separate budgets are maintained for the A and B arms of the experiment.
RIS
TY - CPAPER TI - Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments AU - Guillaume W. Basse AU - Hossein Azari Soufiani AU - Diane Lambert BT - Proceedings of the 19th International Conference on Artificial Intelligence and Statistics DA - 2016/05/02 ED - Arthur Gretton ED - Christian C. Robert ID - pmlr-v51-basse16b PB - PMLR DP - Proceedings of Machine Learning Research VL - 51 SP - 1412 EP - 1420 L1 - http://proceedings.mlr.press/v51/basse16b.pdf UR - https://proceedings.mlr.press/v51/basse16b.html AB - Buyers (e.g., advertisers) often have limited financial or processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling, and any change may affect what buyers pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is an unbiased estimator when all the bidders in the same auction are randomly assigned to A or B but the observed treatment effect can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be much reduced if separate budgets are maintained for the A and B arms of the experiment. ER -
APA
Basse, G.W., Azari Soufiani, H. & Lambert, D.. (2016). Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 51:1412-1420 Available from https://proceedings.mlr.press/v51/basse16b.html.

Related Material