Weighted Conformalized Quantile Regression for Active Learning

Niclas Ståhl, Cecilia Sönströd, Ulf Johansson
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:294-311, 2026.

Abstract

A common approach in active learning is to improve models by acquiring samples for which the prediction is uncertain (Ren et al., 2021). Conformal quantile regression seems like a perfect match for estimating such prediction uncertainties, but any acquisition strategy used in the active learning setting would introduce selection and hence, violate the exchangeability assumption on which conformal prediction relies on. This paper addresses the challenge of maintaining valid uncertainty quantification in active learning for regression by utilizing Weighted Conformalized Quantile Regression with a Weighted CV+ approach. This restores predictive guarantees by re-weighting nonconformity scores based on the likelihood ratio between training and test distributions. Experimental results on the California Housing dataset demonstrate that uncertainty-based acquisition strategies accelerate error reduction (MSE), and that the weighting to maintain target coverage is required in order to keep the guarantees. Furthermore, the analysis identifies that there is a risk that greedy acquisition functions over-sample regions of irreducible noise, and demonstrates that a stochastic weighted strategy effectively mitigates this risk while balancing manifold exploration with predictive precision.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-stahl26a, title = {Weighted Conformalized Quantile Regression for Active Learning}, author = {St{\aa}hl, Niclas and S{\"o}nstr{\"o}d, Cecilia and Johansson, Ulf}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {294--311}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/stahl26a/stahl26a.pdf}, url = {https://proceedings.mlr.press/v329/stahl26a.html}, abstract = {A common approach in active learning is to improve models by acquiring samples for which the prediction is uncertain (Ren et al., 2021). Conformal quantile regression seems like a perfect match for estimating such prediction uncertainties, but any acquisition strategy used in the active learning setting would introduce selection and hence, violate the exchangeability assumption on which conformal prediction relies on. This paper addresses the challenge of maintaining valid uncertainty quantification in active learning for regression by utilizing Weighted Conformalized Quantile Regression with a Weighted CV+ approach. This restores predictive guarantees by re-weighting nonconformity scores based on the likelihood ratio between training and test distributions. Experimental results on the California Housing dataset demonstrate that uncertainty-based acquisition strategies accelerate error reduction (MSE), and that the weighting to maintain target coverage is required in order to keep the guarantees. Furthermore, the analysis identifies that there is a risk that greedy acquisition functions over-sample regions of irreducible noise, and demonstrates that a stochastic weighted strategy effectively mitigates this risk while balancing manifold exploration with predictive precision.} }
Endnote
%0 Conference Paper %T Weighted Conformalized Quantile Regression for Active Learning %A Niclas Ståhl %A Cecilia Sönströd %A Ulf Johansson %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-stahl26a %I PMLR %P 294--311 %U https://proceedings.mlr.press/v329/stahl26a.html %V 329 %X A common approach in active learning is to improve models by acquiring samples for which the prediction is uncertain (Ren et al., 2021). Conformal quantile regression seems like a perfect match for estimating such prediction uncertainties, but any acquisition strategy used in the active learning setting would introduce selection and hence, violate the exchangeability assumption on which conformal prediction relies on. This paper addresses the challenge of maintaining valid uncertainty quantification in active learning for regression by utilizing Weighted Conformalized Quantile Regression with a Weighted CV+ approach. This restores predictive guarantees by re-weighting nonconformity scores based on the likelihood ratio between training and test distributions. Experimental results on the California Housing dataset demonstrate that uncertainty-based acquisition strategies accelerate error reduction (MSE), and that the weighting to maintain target coverage is required in order to keep the guarantees. Furthermore, the analysis identifies that there is a risk that greedy acquisition functions over-sample regions of irreducible noise, and demonstrates that a stochastic weighted strategy effectively mitigates this risk while balancing manifold exploration with predictive precision.
APA
Ståhl, N., Sönströd, C. & Johansson, U.. (2026). Weighted Conformalized Quantile Regression for Active Learning. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:294-311 Available from https://proceedings.mlr.press/v329/stahl26a.html.

Related Material