When Post-Hoc Calibration Hurts: Platt Scaling and Isotonic Regression on Modern Tabular Models

Valery Manokhin, Daniel Grønhaug
Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 329:378-381, 2026.

Abstract

Post-hoc calibration is routinely applied on the assumption that it cannot degrade a classifier. We test that over 3,150 runs per calibrator: 21 classifiers, 30 binary TabArena-v0.1 tasks, 5-fold cross-validation. Under log-loss, two of the most widely used methods are unreliable: isotonic regression has a mean relative change of +3.22% and improves log-loss in only 43.7% of runs, and Platt scaling improves it in 49.8%, showing no consistent directional advantage. Venn–Abers attains the largest mean reduction (-14.17%) and Beta calibration the highest improvement rate (67.1%). Mean AUC-ROC changes by under 1% throughout and mean accuracy is essentially unaffected except under Pearsonify, so changes in probabilistic quality are largely invisible in discrimination and classification metrics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v329-manokhin26a, title = {When Post-Hoc Calibration Hurts: Platt Scaling and Isotonic Regression on Modern Tabular Models}, author = {Manokhin, Valery and Gr{\o}nhaug, Daniel}, booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {378--381}, year = {2026}, editor = {Ahlberg, Ernst and Johansson, Ulf and Boström, Henrik and Carlevaro, Alberto and Hallberg Szabadváry, Johan and Carlsson, Lars}, volume = {329}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v329/main/assets/manokhin26a/manokhin26a.pdf}, url = {https://proceedings.mlr.press/v329/manokhin26a.html}, abstract = {Post-hoc calibration is routinely applied on the assumption that it cannot degrade a classifier. We test that over 3,150 runs per calibrator: 21 classifiers, 30 binary TabArena-v0.1 tasks, 5-fold cross-validation. Under log-loss, two of the most widely used methods are unreliable: isotonic regression has a mean relative change of +3.22% and improves log-loss in only 43.7% of runs, and Platt scaling improves it in 49.8%, showing no consistent directional advantage. Venn–Abers attains the largest mean reduction (-14.17%) and Beta calibration the highest improvement rate (67.1%). Mean AUC-ROC changes by under 1% throughout and mean accuracy is essentially unaffected except under Pearsonify, so changes in probabilistic quality are largely invisible in discrimination and classification metrics.} }
Endnote
%0 Conference Paper %T When Post-Hoc Calibration Hurts: Platt Scaling and Isotonic Regression on Modern Tabular Models %A Valery Manokhin %A Daniel Grønhaug %B Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2026 %E Ernst Ahlberg %E Ulf Johansson %E Henrik Boström %E Alberto Carlevaro %E Johan Hallberg Szabadváry %E Lars Carlsson %F pmlr-v329-manokhin26a %I PMLR %P 378--381 %U https://proceedings.mlr.press/v329/manokhin26a.html %V 329 %X Post-hoc calibration is routinely applied on the assumption that it cannot degrade a classifier. We test that over 3,150 runs per calibrator: 21 classifiers, 30 binary TabArena-v0.1 tasks, 5-fold cross-validation. Under log-loss, two of the most widely used methods are unreliable: isotonic regression has a mean relative change of +3.22% and improves log-loss in only 43.7% of runs, and Platt scaling improves it in 49.8%, showing no consistent directional advantage. Venn–Abers attains the largest mean reduction (-14.17%) and Beta calibration the highest improvement rate (67.1%). Mean AUC-ROC changes by under 1% throughout and mean accuracy is essentially unaffected except under Pearsonify, so changes in probabilistic quality are largely invisible in discrimination and classification metrics.
APA
Manokhin, V. & Grønhaug, D.. (2026). When Post-Hoc Calibration Hurts: Platt Scaling and Isotonic Regression on Modern Tabular Models. Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 329:378-381 Available from https://proceedings.mlr.press/v329/manokhin26a.html.

Related Material