CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

Mohammad Anas Jawad, Cornelia Caragea
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:50921-50944, 2026.

Abstract

Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model’s behavioral robustness to irrelevant or misleading information. In this paper, we argue that a model’s true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model’s susceptibility to distraction. CaliDist quantifies how an LLM’s predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model’s initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average—a relative improvement of 70%—demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/m-anas-j/CaliDist.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-jawad26a, title = {{C}ali{D}ist: Calibrating Large Language Models via Behavioral Robustness to Distraction}, author = {Jawad, Mohammad Anas and Caragea, Cornelia}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {50921--50944}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/jawad26a/jawad26a.pdf}, url = {https://proceedings.mlr.press/v306/jawad26a.html}, abstract = {Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model’s behavioral robustness to irrelevant or misleading information. In this paper, we argue that a model’s true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model’s susceptibility to distraction. CaliDist quantifies how an LLM’s predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model’s initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average—a relative improvement of 70%—demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/m-anas-j/CaliDist.} }
Endnote
%0 Conference Paper %T CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction %A Mohammad Anas Jawad %A Cornelia Caragea %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-jawad26a %I PMLR %P 50921--50944 %U https://proceedings.mlr.press/v306/jawad26a.html %V 306 %X Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model’s behavioral robustness to irrelevant or misleading information. In this paper, we argue that a model’s true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model’s susceptibility to distraction. CaliDist quantifies how an LLM’s predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model’s initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average—a relative improvement of 70%—demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/m-anas-j/CaliDist.
APA
Jawad, M.A. & Caragea, C.. (2026). CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:50921-50944 Available from https://proceedings.mlr.press/v306/jawad26a.html.

Related Material