Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge

Hamid Dadkhahi, Firas Trabelsi, Parker Riley, Juraj Juraska, Mehdi Mirzazadeh
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:22263-22285, 2026.

Abstract

Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate $n$ independent thinking–rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley–Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-dadkhahi26a, title = {Distribution-Calibrated Inference Time Compute for Thinking {LLM}-as-a-Judge}, author = {Dadkhahi, Hamid and Trabelsi, Firas and Riley, Parker and Juraska, Juraj and Mirzazadeh, Mehdi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {22263--22285}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/dadkhahi26a/dadkhahi26a.pdf}, url = {https://proceedings.mlr.press/v306/dadkhahi26a.html}, abstract = {Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate $n$ independent thinking–rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley–Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.} }
Endnote
%0 Conference Paper %T Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge %A Hamid Dadkhahi %A Firas Trabelsi %A Parker Riley %A Juraj Juraska %A Mehdi Mirzazadeh %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-dadkhahi26a %I PMLR %P 22263--22285 %U https://proceedings.mlr.press/v306/dadkhahi26a.html %V 306 %X Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate $n$ independent thinking–rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley–Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.
APA
Dadkhahi, H., Trabelsi, F., Riley, P., Juraska, J. & Mirzazadeh, M.. (2026). Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:22263-22285 Available from https://proceedings.mlr.press/v306/dadkhahi26a.html.

Related Material