Familywise Error Rate Control by Interactive Unmasking

Boyan Duan, Aaditya Ramdas, Larry Wasserman
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2720-2729, 2020.

Abstract

We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows much flexibility: a human (or a computer program acting on the human’s behalf) may adaptively guide the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere to a particular protocol of masking and unmasking. We demonstrate via numerical experiments the power of our test under structured non-nulls, and then explore new forms of masking.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-duan20d, title = {Familywise Error Rate Control by Interactive Unmasking}, author = {Duan, Boyan and Ramdas, Aaditya and Wasserman, Larry}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {2720--2729}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/duan20d/duan20d.pdf}, url = {https://proceedings.mlr.press/v119/duan20d.html}, abstract = {We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows much flexibility: a human (or a computer program acting on the human’s behalf) may adaptively guide the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere to a particular protocol of masking and unmasking. We demonstrate via numerical experiments the power of our test under structured non-nulls, and then explore new forms of masking.} }
Endnote
%0 Conference Paper %T Familywise Error Rate Control by Interactive Unmasking %A Boyan Duan %A Aaditya Ramdas %A Larry Wasserman %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-duan20d %I PMLR %P 2720--2729 %U https://proceedings.mlr.press/v119/duan20d.html %V 119 %X We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows much flexibility: a human (or a computer program acting on the human’s behalf) may adaptively guide the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere to a particular protocol of masking and unmasking. We demonstrate via numerical experiments the power of our test under structured non-nulls, and then explore new forms of masking.
APA
Duan, B., Ramdas, A. & Wasserman, L.. (2020). Familywise Error Rate Control by Interactive Unmasking. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2720-2729 Available from https://proceedings.mlr.press/v119/duan20d.html.

Related Material