Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling

Tom A. Lamb, Desi R. Ivanova, Philip Torr, Tim G. J. Rudner
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:289-297, 2026.

Abstract

Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-lamb26a, title = { Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling }, author = {Lamb, Tom A. and Ivanova, Desi R. and Torr, Philip and Rudner, Tim G. J.}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {289--297}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/lamb26a/lamb26a.pdf}, url = {https://proceedings.mlr.press/v300/lamb26a.html}, abstract = { Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks. } }
Endnote
%0 Conference Paper %T Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling %A Tom A. Lamb %A Desi R. Ivanova %A Philip Torr %A Tim G. J. Rudner %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-lamb26a %I PMLR %P 289--297 %U https://proceedings.mlr.press/v300/lamb26a.html %V 300 %X Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.
APA
Lamb, T.A., Ivanova, D.R., Torr, P. & Rudner, T.G.J.. (2026). Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:289-297 Available from https://proceedings.mlr.press/v300/lamb26a.html.

Related Material