Conformal prediction or the good old DKW from the 50s?

Chancellor Johnstone, Eugene Ndiaye
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:1103-1118, 2026.

Abstract

Prediction sets based on Conformal inference, the Dvoretzky-Kiefer-Wolfowitz-Massart (DKW) inequality, and classical tolerance-region theory are often discussed as if they are competing for the same job while they are not. Each approach controls a different random quantity. In this paper, we organize the three tools around a single construction: pick a map whose image distribution is known, choose a high-mass subset on that scale, then pull back. CP, DKW and the Beta law of Wilks (1941) differ only in which map and which mass measure they use. However, putting these methods on the empirical CDF-scale puts all three on a common axis. With this, two simple inclusions appear. Ordinary CP intervals sit inside DKW intervals at the same nominal confidence level, but the two procedures deliver different guarantees. When the conformal procedure is recalibrated through the Beta law to deliver the same training-conditional (PAC) guarantee as DKW, the conformal interval is still contained in the DKW one. DKW controls the whole empirical path, whereas CP with the Beta law controls one chosen order-statistic interval. We trace the line from the industrial quality control of Shewhart (1931) through the finite-sample tolerance theory of Wilks (1941) to the conformal formulation of Gammerman et al. (1998) and Vovk et al. (2005).

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-johnstone26a, title = {Conformal prediction or the good old DKW from the 50s?}, author = {Johnstone, Chancellor and Ndiaye, Eugene}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {1103--1118}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/johnstone26a/johnstone26a.pdf}, url = {https://proceedings.mlr.press/v329/johnstone26a.html}, abstract = {Prediction sets based on Conformal inference, the Dvoretzky-Kiefer-Wolfowitz-Massart (DKW) inequality, and classical tolerance-region theory are often discussed as if they are competing for the same job while they are not. Each approach controls a different random quantity. In this paper, we organize the three tools around a single construction: pick a map whose image distribution is known, choose a high-mass subset on that scale, then pull back. CP, DKW and the Beta law of Wilks (1941) differ only in which map and which mass measure they use. However, putting these methods on the empirical CDF-scale puts all three on a common axis. With this, two simple inclusions appear. Ordinary CP intervals sit inside DKW intervals at the same nominal confidence level, but the two procedures deliver different guarantees. When the conformal procedure is recalibrated through the Beta law to deliver the same training-conditional (PAC) guarantee as DKW, the conformal interval is still contained in the DKW one. DKW controls the whole empirical path, whereas CP with the Beta law controls one chosen order-statistic interval. We trace the line from the industrial quality control of Shewhart (1931) through the finite-sample tolerance theory of Wilks (1941) to the conformal formulation of Gammerman et al. (1998) and Vovk et al. (2005).} }
Endnote
%0 Conference Paper %T Conformal prediction or the good old DKW from the 50s? %A Chancellor Johnstone %A Eugene Ndiaye %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-johnstone26a %I PMLR %P 1103--1118 %U https://proceedings.mlr.press/v329/johnstone26a.html %V 329 %X Prediction sets based on Conformal inference, the Dvoretzky-Kiefer-Wolfowitz-Massart (DKW) inequality, and classical tolerance-region theory are often discussed as if they are competing for the same job while they are not. Each approach controls a different random quantity. In this paper, we organize the three tools around a single construction: pick a map whose image distribution is known, choose a high-mass subset on that scale, then pull back. CP, DKW and the Beta law of Wilks (1941) differ only in which map and which mass measure they use. However, putting these methods on the empirical CDF-scale puts all three on a common axis. With this, two simple inclusions appear. Ordinary CP intervals sit inside DKW intervals at the same nominal confidence level, but the two procedures deliver different guarantees. When the conformal procedure is recalibrated through the Beta law to deliver the same training-conditional (PAC) guarantee as DKW, the conformal interval is still contained in the DKW one. DKW controls the whole empirical path, whereas CP with the Beta law controls one chosen order-statistic interval. We trace the line from the industrial quality control of Shewhart (1931) through the finite-sample tolerance theory of Wilks (1941) to the conformal formulation of Gammerman et al. (1998) and Vovk et al. (2005).
APA
Johnstone, C. & Ndiaye, E.. (2026). Conformal prediction or the good old DKW from the 50s?. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:1103-1118 Available from https://proceedings.mlr.press/v329/johnstone26a.html.

Related Material