Calibrating Without Labels: Source-Free Conformal Prediction Using Pseudo-Labels

Shachar Angelman, Rotem Nizhar, Jacob Goldberger
Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:63-81, 2025.

Abstract

We address the problem of conformal prediction (CP) in the challenging setting of source-free domain adaptation (SFDA), where models must be calibrated using only unlabeled data from the target domain. Existing CP methods for domain shift rely heavily on labeled source data and importance weighting (IW), but we demonstrate that these approaches perform poorly in practice, even when source labels are available. As an alternative, we propose Source-Free Conformal Prediction (SFCP), a simple and effective method that replaces the unavailable target labels with pseudo-labels generated by the source model. We show both theoretically and empirically that, despite their inherent noise, these pseudo-labels can be reliably used to estimate conformal thresholds. Our method requires no access to source data and no hyperparameter tuning, making it particularly suitable for real-world SFDA scenarios. Experiments across more than 100 domain shifts demonstrate that SFCP achieves coverage levels comparable to oracle CP while consistently outperforming IWbased methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v266-angelman25a, title = {Calibrating Without Labels: Source-Free Conformal Prediction Using Pseudo-Labels}, author = {Angelman, Shachar and Nizhar, Rotem and Goldberger, Jacob}, booktitle = {Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {63--81}, year = {2025}, editor = {Nguyen, Khuong An and Luo, Zhiyuan and Papadopoulos, Harris and Löfström, Tuwe and Carlsson, Lars and Boström, Henrik}, volume = {266}, series = {Proceedings of Machine Learning Research}, month = {10--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v266/main/assets/angelman25a/angelman25a.pdf}, url = {https://proceedings.mlr.press/v266/angelman25a.html}, abstract = {We address the problem of conformal prediction (CP) in the challenging setting of source-free domain adaptation (SFDA), where models must be calibrated using only unlabeled data from the target domain. Existing CP methods for domain shift rely heavily on labeled source data and importance weighting (IW), but we demonstrate that these approaches perform poorly in practice, even when source labels are available. As an alternative, we propose Source-Free Conformal Prediction (SFCP), a simple and effective method that replaces the unavailable target labels with pseudo-labels generated by the source model. We show both theoretically and empirically that, despite their inherent noise, these pseudo-labels can be reliably used to estimate conformal thresholds. Our method requires no access to source data and no hyperparameter tuning, making it particularly suitable for real-world SFDA scenarios. Experiments across more than 100 domain shifts demonstrate that SFCP achieves coverage levels comparable to oracle CP while consistently outperforming IWbased methods.} }
Endnote
%0 Conference Paper %T Calibrating Without Labels: Source-Free Conformal Prediction Using Pseudo-Labels %A Shachar Angelman %A Rotem Nizhar %A Jacob Goldberger %B Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2025 %E Khuong An Nguyen %E Zhiyuan Luo %E Harris Papadopoulos %E Tuwe Löfström %E Lars Carlsson %E Henrik Boström %F pmlr-v266-angelman25a %I PMLR %P 63--81 %U https://proceedings.mlr.press/v266/angelman25a.html %V 266 %X We address the problem of conformal prediction (CP) in the challenging setting of source-free domain adaptation (SFDA), where models must be calibrated using only unlabeled data from the target domain. Existing CP methods for domain shift rely heavily on labeled source data and importance weighting (IW), but we demonstrate that these approaches perform poorly in practice, even when source labels are available. As an alternative, we propose Source-Free Conformal Prediction (SFCP), a simple and effective method that replaces the unavailable target labels with pseudo-labels generated by the source model. We show both theoretically and empirically that, despite their inherent noise, these pseudo-labels can be reliably used to estimate conformal thresholds. Our method requires no access to source data and no hyperparameter tuning, making it particularly suitable for real-world SFDA scenarios. Experiments across more than 100 domain shifts demonstrate that SFCP achieves coverage levels comparable to oracle CP while consistently outperforming IWbased methods.
APA
Angelman, S., Nizhar, R. & Goldberger, J.. (2025). Calibrating Without Labels: Source-Free Conformal Prediction Using Pseudo-Labels. Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 266:63-81 Available from https://proceedings.mlr.press/v266/angelman25a.html.

Related Material