Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation

Qining Zhang, Tanner Fiez, Yi Liu, Wenyang Liu
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2107-2115, 2026.

Abstract

A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment’s quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-zhang26c, title = { Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation }, author = {Zhang, Qining and Fiez, Tanner and Liu, Yi and Liu, Wenyang}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2107--2115}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/zhang26c/zhang26c.pdf}, url = {https://proceedings.mlr.press/v300/zhang26c.html}, abstract = { A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment’s quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness. } }
Endnote
%0 Conference Paper %T Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation %A Qining Zhang %A Tanner Fiez %A Yi Liu %A Wenyang Liu %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-zhang26c %I PMLR %P 2107--2115 %U https://proceedings.mlr.press/v300/zhang26c.html %V 300 %X A/B tests in online experiments face statistical power challenges when testing multiple candidates simultaneously, while adaptive experimental designs (AED) alone fall short in inferring experiment statistics such as the average treatment effect, especially with many metrics (e.g., revenue, safety) and heterogeneous variances. This paper proposes a fixed-budget multi-metric AED framework with a two-phase structure: an adaptive exploration phase to identify the best treatment, and a validation phase with an A/B test to verify the treatment’s quality and infer statistics. We propose SHRVar, which generalizes sequential halving (SH) with a novel relative-variance-based sampling and an elimination strategy built on reward z values. It achieves a provable error probability that decreases exponentially, where the exponent H3 generalizes the complexity measure for SH and SHVar with homogeneous and heterogeneous variances, respectively. Numerical experiments demonstrate its performance and robustness.
APA
Zhang, Q., Fiez, T., Liu, Y. & Liu, W.. (2026). Multi-Metric Adaptive Experimental Design Under a Fixed Budget with Validation . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2107-2115 Available from https://proceedings.mlr.press/v300/zhang26c.html.

Related Material