Meta-Cal: Well-controlled Post-hoc Calibration by Ranking

Xingchen Ma, Matthew B. Blaschko
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:7235-7245, 2021.

Abstract

In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a model by learning a calibration map. Existing approaches mostly focus on constructing calibration maps with low calibration errors, however, this quality is inadequate for a calibrator being useful. In this paper, we introduce two constraints that are worth consideration in designing a calibration map for post-hoc calibration. Then we present Meta-Cal, which is built from a base calibrator and a ranking model. Under some mild assumptions, two high-probability bounds are given with respect to these constraints. Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular network architectures show our proposed method significantly outperforms the current state of the art for post-hoc multi-class classification calibration.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-ma21a, title = {Meta-Cal: Well-controlled Post-hoc Calibration by Ranking}, author = {Ma, Xingchen and Blaschko, Matthew B.}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {7235--7245}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/ma21a/ma21a.pdf}, url = {https://proceedings.mlr.press/v139/ma21a.html}, abstract = {In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a model by learning a calibration map. Existing approaches mostly focus on constructing calibration maps with low calibration errors, however, this quality is inadequate for a calibrator being useful. In this paper, we introduce two constraints that are worth consideration in designing a calibration map for post-hoc calibration. Then we present Meta-Cal, which is built from a base calibrator and a ranking model. Under some mild assumptions, two high-probability bounds are given with respect to these constraints. Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular network architectures show our proposed method significantly outperforms the current state of the art for post-hoc multi-class classification calibration.} }
Endnote
%0 Conference Paper %T Meta-Cal: Well-controlled Post-hoc Calibration by Ranking %A Xingchen Ma %A Matthew B. Blaschko %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-ma21a %I PMLR %P 7235--7245 %U https://proceedings.mlr.press/v139/ma21a.html %V 139 %X In many applications, it is desirable that a classifier not only makes accurate predictions, but also outputs calibrated posterior probabilities. However, many existing classifiers, especially deep neural network classifiers, tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a model by learning a calibration map. Existing approaches mostly focus on constructing calibration maps with low calibration errors, however, this quality is inadequate for a calibrator being useful. In this paper, we introduce two constraints that are worth consideration in designing a calibration map for post-hoc calibration. Then we present Meta-Cal, which is built from a base calibrator and a ranking model. Under some mild assumptions, two high-probability bounds are given with respect to these constraints. Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular network architectures show our proposed method significantly outperforms the current state of the art for post-hoc multi-class classification calibration.
APA
Ma, X. & Blaschko, M.B.. (2021). Meta-Cal: Well-controlled Post-hoc Calibration by Ranking. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:7235-7245 Available from https://proceedings.mlr.press/v139/ma21a.html.

Related Material