Estimating the Number and Effect Sizes of Non-null Hypotheses

Jennifer Brennan, Ramya Korlakai Vinayak, Kevin Jamieson
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1123-1133, 2020.

Abstract

We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a future experiment. We prove that this simple and computationally efficient estimator enjoys a number of favorable theoretical properties, and demonstrate its effectiveness on data from a gene knockout experiment on influenza inhibition in Drosophila.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-brennan20a, title = {Estimating the Number and Effect Sizes of Non-null Hypotheses}, author = {Brennan, Jennifer and Vinayak, Ramya Korlakai and Jamieson, Kevin}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1123--1133}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/brennan20a/brennan20a.pdf}, url = {https://proceedings.mlr.press/v119/brennan20a.html}, abstract = {We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a future experiment. We prove that this simple and computationally efficient estimator enjoys a number of favorable theoretical properties, and demonstrate its effectiveness on data from a gene knockout experiment on influenza inhibition in Drosophila.} }
Endnote
%0 Conference Paper %T Estimating the Number and Effect Sizes of Non-null Hypotheses %A Jennifer Brennan %A Ramya Korlakai Vinayak %A Kevin Jamieson %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-brennan20a %I PMLR %P 1123--1133 %U https://proceedings.mlr.press/v119/brennan20a.html %V 119 %X We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a future experiment. We prove that this simple and computationally efficient estimator enjoys a number of favorable theoretical properties, and demonstrate its effectiveness on data from a gene knockout experiment on influenza inhibition in Drosophila.
APA
Brennan, J., Vinayak, R.K. & Jamieson, K.. (2020). Estimating the Number and Effect Sizes of Non-null Hypotheses. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1123-1133 Available from https://proceedings.mlr.press/v119/brennan20a.html.

Related Material