Few-Shot Conformal Prediction with Auxiliary Tasks

Adam Fisch, Tal Schuster, Tommi Jaakkola, Dr.Regina Barzilay
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:3329-3339, 2021.

Abstract

We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-fisch21a, title = {Few-Shot Conformal Prediction with Auxiliary Tasks}, author = {Fisch, Adam and Schuster, Tal and Jaakkola, Tommi and Barzilay, Dr.Regina}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {3329--3339}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/fisch21a/fisch21a.pdf}, url = {https://proceedings.mlr.press/v139/fisch21a.html}, abstract = {We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.} }
Endnote
%0 Conference Paper %T Few-Shot Conformal Prediction with Auxiliary Tasks %A Adam Fisch %A Tal Schuster %A Tommi Jaakkola %A Dr.Regina Barzilay %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-fisch21a %I PMLR %P 3329--3339 %U https://proceedings.mlr.press/v139/fisch21a.html %V 139 %X We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
APA
Fisch, A., Schuster, T., Jaakkola, T. & Barzilay, D.. (2021). Few-Shot Conformal Prediction with Auxiliary Tasks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:3329-3339 Available from https://proceedings.mlr.press/v139/fisch21a.html.

Related Material