Sequential Multiple Hypothesis Testing with Type I Error Control

Alan Malek, Sumeet Katariya, Yinlam Chow, Mohammad Ghavamzadeh
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:1468-1476, 2017.

Abstract

This work studies multiple hypothesis testing in the setting when we obtain data sequentially and may choose when to stop sampling. We summarize the notion of a sequential p-value (one that can be continually updated and still maintain a type I error guarantee) and provide several examples from the literature. This tool allows us to convert step-up or step-down multiple hypothesis testing procedures in the fixed-horizon setting (which includes Benjamini-Hochberg, Holm, and Bonferroni) into sequential versions that allow the statistician to reject a hypothesis as soon as the sequential p-value reaches a threshold. We show that if the original procedure has a type I error guarantee in a certain family (including FDR and FWER), then the sequential conversion inherits an analogous guarantee. The conversion also allows for allocating samples in a data-dependent way, and we provide simulated experiments demonstrating an increased number of rejections when compared to the fixed-horizon setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v54-malek17a, title = {{Sequential Multiple Hypothesis Testing with Type I Error Control}}, author = {Malek, Alan and Katariya, Sumeet and Chow, Yinlam and Ghavamzadeh, Mohammad}, booktitle = {Proceedings of the 20th International Conference on Artificial Intelligence and Statistics}, pages = {1468--1476}, year = {2017}, editor = {Singh, Aarti and Zhu, Jerry}, volume = {54}, series = {Proceedings of Machine Learning Research}, month = {20--22 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v54/malek17a/malek17a.pdf}, url = {https://proceedings.mlr.press/v54/malek17a.html}, abstract = {This work studies multiple hypothesis testing in the setting when we obtain data sequentially and may choose when to stop sampling. We summarize the notion of a sequential p-value (one that can be continually updated and still maintain a type I error guarantee) and provide several examples from the literature. This tool allows us to convert step-up or step-down multiple hypothesis testing procedures in the fixed-horizon setting (which includes Benjamini-Hochberg, Holm, and Bonferroni) into sequential versions that allow the statistician to reject a hypothesis as soon as the sequential p-value reaches a threshold. We show that if the original procedure has a type I error guarantee in a certain family (including FDR and FWER), then the sequential conversion inherits an analogous guarantee. The conversion also allows for allocating samples in a data-dependent way, and we provide simulated experiments demonstrating an increased number of rejections when compared to the fixed-horizon setting.} }
Endnote
%0 Conference Paper %T Sequential Multiple Hypothesis Testing with Type I Error Control %A Alan Malek %A Sumeet Katariya %A Yinlam Chow %A Mohammad Ghavamzadeh %B Proceedings of the 20th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2017 %E Aarti Singh %E Jerry Zhu %F pmlr-v54-malek17a %I PMLR %P 1468--1476 %U https://proceedings.mlr.press/v54/malek17a.html %V 54 %X This work studies multiple hypothesis testing in the setting when we obtain data sequentially and may choose when to stop sampling. We summarize the notion of a sequential p-value (one that can be continually updated and still maintain a type I error guarantee) and provide several examples from the literature. This tool allows us to convert step-up or step-down multiple hypothesis testing procedures in the fixed-horizon setting (which includes Benjamini-Hochberg, Holm, and Bonferroni) into sequential versions that allow the statistician to reject a hypothesis as soon as the sequential p-value reaches a threshold. We show that if the original procedure has a type I error guarantee in a certain family (including FDR and FWER), then the sequential conversion inherits an analogous guarantee. The conversion also allows for allocating samples in a data-dependent way, and we provide simulated experiments demonstrating an increased number of rejections when compared to the fixed-horizon setting.
APA
Malek, A., Katariya, S., Chow, Y. & Ghavamzadeh, M.. (2017). Sequential Multiple Hypothesis Testing with Type I Error Control. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 54:1468-1476 Available from https://proceedings.mlr.press/v54/malek17a.html.

Related Material