SAFFRON: an Adaptive Algorithm for Online Control of the False Discovery Rate
[edit]
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:42864294, 2018.
Abstract
In the online false discovery rate (FDR) problem, one observes a possibly infinite sequence of $p$values $P_1,P_2,…$, each testing a different null hypothesis, and an algorithm must pick a sequence of rejection thresholds $\alpha_1,\alpha_2,…$ in an online fashion, effectively rejecting the $k$th null hypothesis whenever $P_k \leq \alpha_k$. Importantly, $\alpha_k$ must be a function of the past, and cannot depend on $P_k$ or any of the later unseen $p$values, and must be chosen to guarantee that for any time $t$, the FDR up to time $t$ is less than some predetermined quantity $\alpha \in (0,1)$. In this work, we present a powerful new framework for online FDR control that we refer to as “SAFFRON”. Like older alphainvesting algorithms, SAFFRON starts off with an error budget (called alphawealth) that it intelligently allocates to different tests over time, earning back some alphawealth whenever it makes a new discovery. However, unlike older methods, SAFFRON’s threshold sequence is based on a novel estimate of the alpha fraction that it allocates to true null hypotheses. In the offline setting, algorithms that employ an estimate of the proportion of true nulls are called “adaptive”, hence SAFFRON can be seen as an online analogue of the offline StoreyBH adaptive procedure. Just as StoreyBH is typically more powerful than the BenjaminiHochberg (BH) procedure under independence, we demonstrate that SAFFRON is also more powerful than its nonadaptive counterparts such as LORD.
Related Material


