Online Control of the False Coverage Rate and False Sign Rate

Asaf Weinstein, Aaditya Ramdas
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10193-10202, 2020.

Abstract

The reproducibility debate has caused a renewed interest in changing how one reports uncertainty, from $p$-value for testing a null hypothesis to a confidence interval (CI) for the corresponding parameter. When CIs for multiple selected parameters are being reported, the analog of the false discovery rate (FDR) is the false coverage rate (FCR), which is the expected ratio of number of reported CIs failing to cover their respective parameters to the total number of reported CIs. Here, we consider the general problem of FCR control in the online setting, where one encounters an infinite sequence of fixed unknown parameters ordered by time. We propose a novel solution to the problem which only requires the scientist to be able to construct marginal CIs. As special cases, our framework yields algorithms for online FDR control and online sign-classification procedures that control the false sign rate (FSR). All of our methodology applies equally well to prediction intervals, having particular implications for selective conformal inference.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-weinstein20a, title = {Online Control of the False Coverage Rate and False Sign Rate}, author = {Weinstein, Asaf and Ramdas, Aaditya}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10193--10202}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/weinstein20a/weinstein20a.pdf}, url = {https://proceedings.mlr.press/v119/weinstein20a.html}, abstract = {The reproducibility debate has caused a renewed interest in changing how one reports uncertainty, from $p$-value for testing a null hypothesis to a confidence interval (CI) for the corresponding parameter. When CIs for multiple selected parameters are being reported, the analog of the false discovery rate (FDR) is the false coverage rate (FCR), which is the expected ratio of number of reported CIs failing to cover their respective parameters to the total number of reported CIs. Here, we consider the general problem of FCR control in the online setting, where one encounters an infinite sequence of fixed unknown parameters ordered by time. We propose a novel solution to the problem which only requires the scientist to be able to construct marginal CIs. As special cases, our framework yields algorithms for online FDR control and online sign-classification procedures that control the false sign rate (FSR). All of our methodology applies equally well to prediction intervals, having particular implications for selective conformal inference.} }
Endnote
%0 Conference Paper %T Online Control of the False Coverage Rate and False Sign Rate %A Asaf Weinstein %A Aaditya Ramdas %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-weinstein20a %I PMLR %P 10193--10202 %U https://proceedings.mlr.press/v119/weinstein20a.html %V 119 %X The reproducibility debate has caused a renewed interest in changing how one reports uncertainty, from $p$-value for testing a null hypothesis to a confidence interval (CI) for the corresponding parameter. When CIs for multiple selected parameters are being reported, the analog of the false discovery rate (FDR) is the false coverage rate (FCR), which is the expected ratio of number of reported CIs failing to cover their respective parameters to the total number of reported CIs. Here, we consider the general problem of FCR control in the online setting, where one encounters an infinite sequence of fixed unknown parameters ordered by time. We propose a novel solution to the problem which only requires the scientist to be able to construct marginal CIs. As special cases, our framework yields algorithms for online FDR control and online sign-classification procedures that control the false sign rate (FSR). All of our methodology applies equally well to prediction intervals, having particular implications for selective conformal inference.
APA
Weinstein, A. & Ramdas, A.. (2020). Online Control of the False Coverage Rate and False Sign Rate. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10193-10202 Available from https://proceedings.mlr.press/v119/weinstein20a.html.

Related Material