A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:141853-141886, 2026.

Abstract

Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates—more data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-xu26ay, title = {A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth}, author = {Xu, Mingyuan and Tan, Xinzi and Wu, Jiawei and Zhou, Doudou}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {141853--141886}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xu26ay/xu26ay.pdf}, url = {https://proceedings.mlr.press/v306/xu26ay.html}, abstract = {Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates—more data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.} }
Endnote
%0 Conference Paper %T A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth %A Mingyuan Xu %A Xinzi Tan %A Jiawei Wu %A Doudou Zhou %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xu26ay %I PMLR %P 141853--141886 %U https://proceedings.mlr.press/v306/xu26ay.html %V 306 %X Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates—more data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.
APA
Xu, M., Tan, X., Wu, J. & Zhou, D.. (2026). A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:141853-141886 Available from https://proceedings.mlr.press/v306/xu26ay.html.

Related Material