Calibration of Natural Language Understanding Models with Venn–ABERS Predictors

Patrizio Giovannotti
Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 179:55-71, 2022.

Abstract

Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn–ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval – all while retaining the original model�s predictive accuracy.

Cite this Paper

BibTeX
@InProceedings{pmlr-v179-giovannotti22a, title = { Calibration of Natural Language Understanding Models with Venn–ABERS Predictors}, author = {Giovannotti, Patrizio}, booktitle = {Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {55--71}, year = {2022}, editor = {Johansson, Ulf and Boström, Henrik and An Nguyen, Khuong and Luo, Zhiyuan and Carlsson, Lars}, volume = {179}, series = {Proceedings of Machine Learning Research}, month = {24--26 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v179/giovannotti22a/giovannotti22a.pdf}, url = {https://proceedings.mlr.press/v179/giovannotti22a.html}, abstract = {Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn–ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval – all while retaining the original model�s predictive accuracy. } }
Endnote
%0 Conference Paper %T Calibration of Natural Language Understanding Models with Venn–ABERS Predictors %A Patrizio Giovannotti %B Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2022 %E Ulf Johansson %E Henrik Boström %E Khuong An Nguyen %E Zhiyuan Luo %E Lars Carlsson %F pmlr-v179-giovannotti22a %I PMLR %P 55--71 %U https://proceedings.mlr.press/v179/giovannotti22a.html %V 179 %X Transformers, currently the state-of-the-art in natural language understanding (NLU) tasks, are prone to generate uncalibrated predictions or extreme probabilities, making the process of taking different decisions based on their output relatively difficult. In this paper we propose to build several inductive Venn–ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers. We test their performance over a set of diverse NLU tasks and show that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval – all while retaining the original model�s predictive accuracy.
APA
Giovannotti, P.. (2022). Calibration of Natural Language Understanding Models with Venn–ABERS Predictors. Proceedings of the Eleventh Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 179:55-71 Available from https://proceedings.mlr.press/v179/giovannotti22a.html.

Related Material