Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks

Jiawei Gu, Fengyuan Nie, Hao Tang, Yanpeng Sun
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:37158-37194, 2026.

Abstract

Low-precision arithmetic is pervasive in neural network training and deployment, yet its effect on prediction confidence, not just accuracy, remains unexamined. We show that the softmax function amplifies logit-space quantization errors in an input-dependent manner: confidence distortion scales with the product of precision-dependent error bound $\epsilon$ and logit norm, peaking when the model is confident but not saturated. This explains why identical models report different confidence values across precisions, a phenomenon we term Precision Split. During training, the same mechanism causes gradient underflow: when logit margins exceed a precision-dependent threshold, gradients vanish and samples silently stop contributing to learning. Since logit norm serves as a computable proxy for precision-induced risk, we propose Precision-Aware Confidence Scaling (PACS), which applies sample-adaptive temperature inversely related to this risk, with sub-one-percent overhead and no full-precision computation required. On ImageNet with mixed-precision ResNet-50, PACS reduces Expected Calibration Error from 5.82% to 1.92% while maintaining accuracy, with consistent improvements across architectures, precision formats, and modalities.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gu26j, title = {Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks}, author = {Gu, Jiawei and Nie, Fengyuan and Tang, Hao and Sun, Yanpeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {37158--37194}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gu26j/gu26j.pdf}, url = {https://proceedings.mlr.press/v306/gu26j.html}, abstract = {Low-precision arithmetic is pervasive in neural network training and deployment, yet its effect on prediction confidence, not just accuracy, remains unexamined. We show that the softmax function amplifies logit-space quantization errors in an input-dependent manner: confidence distortion scales with the product of precision-dependent error bound $\epsilon$ and logit norm, peaking when the model is confident but not saturated. This explains why identical models report different confidence values across precisions, a phenomenon we term Precision Split. During training, the same mechanism causes gradient underflow: when logit margins exceed a precision-dependent threshold, gradients vanish and samples silently stop contributing to learning. Since logit norm serves as a computable proxy for precision-induced risk, we propose Precision-Aware Confidence Scaling (PACS), which applies sample-adaptive temperature inversely related to this risk, with sub-one-percent overhead and no full-precision computation required. On ImageNet with mixed-precision ResNet-50, PACS reduces Expected Calibration Error from 5.82% to 1.92% while maintaining accuracy, with consistent improvements across architectures, precision formats, and modalities.} }
Endnote
%0 Conference Paper %T Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks %A Jiawei Gu %A Fengyuan Nie %A Hao Tang %A Yanpeng Sun %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gu26j %I PMLR %P 37158--37194 %U https://proceedings.mlr.press/v306/gu26j.html %V 306 %X Low-precision arithmetic is pervasive in neural network training and deployment, yet its effect on prediction confidence, not just accuracy, remains unexamined. We show that the softmax function amplifies logit-space quantization errors in an input-dependent manner: confidence distortion scales with the product of precision-dependent error bound $\epsilon$ and logit norm, peaking when the model is confident but not saturated. This explains why identical models report different confidence values across precisions, a phenomenon we term Precision Split. During training, the same mechanism causes gradient underflow: when logit margins exceed a precision-dependent threshold, gradients vanish and samples silently stop contributing to learning. Since logit norm serves as a computable proxy for precision-induced risk, we propose Precision-Aware Confidence Scaling (PACS), which applies sample-adaptive temperature inversely related to this risk, with sub-one-percent overhead and no full-precision computation required. On ImageNet with mixed-precision ResNet-50, PACS reduces Expected Calibration Error from 5.82% to 1.92% while maintaining accuracy, with consistent improvements across architectures, precision formats, and modalities.
APA
Gu, J., Nie, F., Tang, H. & Sun, Y.. (2026). Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:37158-37194 Available from https://proceedings.mlr.press/v306/gu26j.html.

Related Material