CRPS-Optimal Binning for Univariate Conformal Regression

Paolo Toccaceli
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:249-278, 2026.

Abstract

We propose a non-parametric, model-free method for conditional distribution estimation based on partitioning covariate-sorted observations into contiguous bins and using the within-bin empirical CDF as the predictive distribution. Bin boundaries are chosen to minimise the total leave-one-out Continuous Ranked Probability Score (LOO-CRPS), which admits a closed-form cost function with O(n2 log n) precomputation and O(n2) storage; the globally optimal K-partition is recovered by a dynamic programme in O(n2K) time. We select K by K-fold cross-validation of test CRPS, which yields a U-shaped criterion with a well-defined minimum. Having selected K* and fitted the full-data partition, we form a conformal prediction set based on CRPS as the nonconformity score, which carries a finite-sample marginal coverage guarantee at any prescribed level $\varepsilon$. The conformal prediction is transductive and data-efficient, as all observations are used for both partitioning and p-value calculation, with no need to reserve a hold-out set. On real-world small-data benchmarks against split-conformal competitors (Gaussian split conformal, CQR, CQR-QRF, CHR, and conformalized isotonic distributional regression), the full-n method produces narrower prediction intervals while maintaining near-nominal coverage; the advantage is not as clear-cut however in a matched-sample comparison restricted to the same training half as the competitors. Similarly, a comparison against QOOB, another transductive competitor, provides mixed results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-toccaceli26a, title = {CRPS-Optimal Binning for Univariate Conformal Regression}, author = {Toccaceli, Paolo}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {249--278}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/toccaceli26a/toccaceli26a.pdf}, url = {https://proceedings.mlr.press/v329/toccaceli26a.html}, abstract = {We propose a non-parametric, model-free method for conditional distribution estimation based on partitioning covariate-sorted observations into contiguous bins and using the within-bin empirical CDF as the predictive distribution. Bin boundaries are chosen to minimise the total leave-one-out Continuous Ranked Probability Score (LOO-CRPS), which admits a closed-form cost function with O(n2 log n) precomputation and O(n2) storage; the globally optimal K-partition is recovered by a dynamic programme in O(n2K) time. We select K by K-fold cross-validation of test CRPS, which yields a U-shaped criterion with a well-defined minimum. Having selected K* and fitted the full-data partition, we form a conformal prediction set based on CRPS as the nonconformity score, which carries a finite-sample marginal coverage guarantee at any prescribed level $\varepsilon$. The conformal prediction is transductive and data-efficient, as all observations are used for both partitioning and p-value calculation, with no need to reserve a hold-out set. On real-world small-data benchmarks against split-conformal competitors (Gaussian split conformal, CQR, CQR-QRF, CHR, and conformalized isotonic distributional regression), the full-n method produces narrower prediction intervals while maintaining near-nominal coverage; the advantage is not as clear-cut however in a matched-sample comparison restricted to the same training half as the competitors. Similarly, a comparison against QOOB, another transductive competitor, provides mixed results.} }
Endnote
%0 Conference Paper %T CRPS-Optimal Binning for Univariate Conformal Regression %A Paolo Toccaceli %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-toccaceli26a %I PMLR %P 249--278 %U https://proceedings.mlr.press/v329/toccaceli26a.html %V 329 %X We propose a non-parametric, model-free method for conditional distribution estimation based on partitioning covariate-sorted observations into contiguous bins and using the within-bin empirical CDF as the predictive distribution. Bin boundaries are chosen to minimise the total leave-one-out Continuous Ranked Probability Score (LOO-CRPS), which admits a closed-form cost function with O(n2 log n) precomputation and O(n2) storage; the globally optimal K-partition is recovered by a dynamic programme in O(n2K) time. We select K by K-fold cross-validation of test CRPS, which yields a U-shaped criterion with a well-defined minimum. Having selected K* and fitted the full-data partition, we form a conformal prediction set based on CRPS as the nonconformity score, which carries a finite-sample marginal coverage guarantee at any prescribed level $\varepsilon$. The conformal prediction is transductive and data-efficient, as all observations are used for both partitioning and p-value calculation, with no need to reserve a hold-out set. On real-world small-data benchmarks against split-conformal competitors (Gaussian split conformal, CQR, CQR-QRF, CHR, and conformalized isotonic distributional regression), the full-n method produces narrower prediction intervals while maintaining near-nominal coverage; the advantage is not as clear-cut however in a matched-sample comparison restricted to the same training half as the competitors. Similarly, a comparison against QOOB, another transductive competitor, provides mixed results.
APA
Toccaceli, P.. (2026). CRPS-Optimal Binning for Univariate Conformal Regression. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:249-278 Available from https://proceedings.mlr.press/v329/toccaceli26a.html.

Related Material