LLMs Judging LLMs: A Simplex Perspective

Patrick Vossler, Fan Xia, Yifan Mai, Adarsh Subbaswamy, Jean Feng
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:1603-1611, 2026.

Abstract

Given the challenge of automatically evaluating free-form outputs from large language models (LLMs), a common solution is to use LLMs themselves as judges, without any gold-standard scores. Implicitly, this practice accounts for only sampling variability (aleatoric uncertainty) and ignores uncertainty about judge quality (epistemic uncertainty). While this is justified if judges are perfectly accurate, it is unclear when such an approach is theoretically valid and practically robust. We study these questions for the task of ranking LLM candidates from a novel geometric perspective: for $M$-level scoring systems, both LLM judges and candidates can be represented as points on an $(M-1)$-dimensional probability simplex, where geometric concepts (e.g., triangle areas)correspond to key ranking concepts. This perspective yields intuitive theoretical conditions and visual proofs for when rankings are identifiable; for instance, we provide a formal basis for the "folk wisdom" that LLM judges are more effective for two-level scoring ($M=2$) than multi-level scoring ($M>2$). Using this geometric intuition, we design Bayesian priors that encode epistemic uncertainty and vary the priors to conduct sensitivity analyses. Experiments on LLM benchmarks show that rankings based solely on LLM judges are robust in many but not all datasets, underscoring both their widespread success and the need for caution. Our Bayesian method achieves substantially higher coverage rates than existing procedures by modeling epistemic uncertainty.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-vossler26a, title = { LLMs Judging LLMs: A Simplex Perspective }, author = {Vossler, Patrick and Xia, Fan and Mai, Yifan and Subbaswamy, Adarsh and Feng, Jean}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {1603--1611}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/vossler26a/vossler26a.pdf}, url = {https://proceedings.mlr.press/v300/vossler26a.html}, abstract = { Given the challenge of automatically evaluating free-form outputs from large language models (LLMs), a common solution is to use LLMs themselves as judges, without any gold-standard scores. Implicitly, this practice accounts for only sampling variability (aleatoric uncertainty) and ignores uncertainty about judge quality (epistemic uncertainty). While this is justified if judges are perfectly accurate, it is unclear when such an approach is theoretically valid and practically robust. We study these questions for the task of ranking LLM candidates from a novel geometric perspective: for $M$-level scoring systems, both LLM judges and candidates can be represented as points on an $(M-1)$-dimensional probability simplex, where geometric concepts (e.g., triangle areas)correspond to key ranking concepts. This perspective yields intuitive theoretical conditions and visual proofs for when rankings are identifiable; for instance, we provide a formal basis for the "folk wisdom" that LLM judges are more effective for two-level scoring ($M=2$) than multi-level scoring ($M>2$). Using this geometric intuition, we design Bayesian priors that encode epistemic uncertainty and vary the priors to conduct sensitivity analyses. Experiments on LLM benchmarks show that rankings based solely on LLM judges are robust in many but not all datasets, underscoring both their widespread success and the need for caution. Our Bayesian method achieves substantially higher coverage rates than existing procedures by modeling epistemic uncertainty. } }
Endnote
%0 Conference Paper %T LLMs Judging LLMs: A Simplex Perspective %A Patrick Vossler %A Fan Xia %A Yifan Mai %A Adarsh Subbaswamy %A Jean Feng %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-vossler26a %I PMLR %P 1603--1611 %U https://proceedings.mlr.press/v300/vossler26a.html %V 300 %X Given the challenge of automatically evaluating free-form outputs from large language models (LLMs), a common solution is to use LLMs themselves as judges, without any gold-standard scores. Implicitly, this practice accounts for only sampling variability (aleatoric uncertainty) and ignores uncertainty about judge quality (epistemic uncertainty). While this is justified if judges are perfectly accurate, it is unclear when such an approach is theoretically valid and practically robust. We study these questions for the task of ranking LLM candidates from a novel geometric perspective: for $M$-level scoring systems, both LLM judges and candidates can be represented as points on an $(M-1)$-dimensional probability simplex, where geometric concepts (e.g., triangle areas)correspond to key ranking concepts. This perspective yields intuitive theoretical conditions and visual proofs for when rankings are identifiable; for instance, we provide a formal basis for the "folk wisdom" that LLM judges are more effective for two-level scoring ($M=2$) than multi-level scoring ($M>2$). Using this geometric intuition, we design Bayesian priors that encode epistemic uncertainty and vary the priors to conduct sensitivity analyses. Experiments on LLM benchmarks show that rankings based solely on LLM judges are robust in many but not all datasets, underscoring both their widespread success and the need for caution. Our Bayesian method achieves substantially higher coverage rates than existing procedures by modeling epistemic uncertainty.
APA
Vossler, P., Xia, F., Mai, Y., Subbaswamy, A. & Feng, J.. (2026). LLMs Judging LLMs: A Simplex Perspective . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:1603-1611 Available from https://proceedings.mlr.press/v300/vossler26a.html.

Related Material