Transductive conformal inference with adaptive scores

Ulysse Gazin, Gilles Blanchard, Etienne Roquain
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:1504-1512, 2024.

Abstract

Conformal inference is a fundamental and versatile tool that provides distribution-free guarantees for many machine learning tasks. We consider the transductive setting, where decisions are made on a test sample of m new points, giving rise to m conformal p-values. While classical results only concern their marginal distribution, we show that their joint distribution follows a Pólya urn model, and establish a concentration inequality for their empirical distribution function. The results hold for arbitrary exchangeable scores, including adaptive ones that can use the covariates of the test+calibration samples at training stage for increased accuracy. We demonstrate the usefulness of these theoretical results through uniform, in-probability guarantees for two machine learning tasks of current interest: interval prediction for transductive transfer learning and novelty detection based on two-class classification.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-gazin24a, title = {Transductive conformal inference with adaptive scores}, author = {Gazin, Ulysse and Blanchard, Gilles and Roquain, Etienne}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {1504--1512}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/gazin24a/gazin24a.pdf}, url = {https://proceedings.mlr.press/v238/gazin24a.html}, abstract = {Conformal inference is a fundamental and versatile tool that provides distribution-free guarantees for many machine learning tasks. We consider the transductive setting, where decisions are made on a test sample of $m$ new points, giving rise to $m$ conformal $p$-values. While classical results only concern their marginal distribution, we show that their joint distribution follows a Pólya urn model, and establish a concentration inequality for their empirical distribution function. The results hold for arbitrary exchangeable scores, including adaptive ones that can use the covariates of the test${+}$calibration samples at training stage for increased accuracy. We demonstrate the usefulness of these theoretical results through uniform, in-probability guarantees for two machine learning tasks of current interest: interval prediction for transductive transfer learning and novelty detection based on two-class classification.} }
Endnote
%0 Conference Paper %T Transductive conformal inference with adaptive scores %A Ulysse Gazin %A Gilles Blanchard %A Etienne Roquain %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-gazin24a %I PMLR %P 1504--1512 %U https://proceedings.mlr.press/v238/gazin24a.html %V 238 %X Conformal inference is a fundamental and versatile tool that provides distribution-free guarantees for many machine learning tasks. We consider the transductive setting, where decisions are made on a test sample of $m$ new points, giving rise to $m$ conformal $p$-values. While classical results only concern their marginal distribution, we show that their joint distribution follows a Pólya urn model, and establish a concentration inequality for their empirical distribution function. The results hold for arbitrary exchangeable scores, including adaptive ones that can use the covariates of the test${+}$calibration samples at training stage for increased accuracy. We demonstrate the usefulness of these theoretical results through uniform, in-probability guarantees for two machine learning tasks of current interest: interval prediction for transductive transfer learning and novelty detection based on two-class classification.
APA
Gazin, U., Blanchard, G. & Roquain, E.. (2024). Transductive conformal inference with adaptive scores. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:1504-1512 Available from https://proceedings.mlr.press/v238/gazin24a.html.

Related Material