Pluralistic Leaderboards

Nika Haghtalab, Ariel D. Procaccia, Han Shao, Serena Lutong Wang, Kunhe Yang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:39356-39375, 2026.

Abstract

Recent leaderboard-based evaluations of large language models aggregate user feedback by fitting a Bradley–Terry model to pairwise comparisons, producing a single global ranking based on a latent quality score. While appealing for its simplicity, this approach is incompatible with heterogeneous preferences: when LLMs are used across diverse tasks and use cases, users who favor fundamentally different model behaviors can be systematically misrepresented when collapsed into a single quality score. To address this issue, we study pluralistic leaderboards that aim to remain stable with respect to heterogeneous user populations. Drawing on ideas from social choice theory, we adapt the notion of local stability, which requires that no model outside the top-$k$ positions is collectively preferred to the top-$k$ set by more than $O(1/k)$ fraction of users. Building on techniques from the social choice literature, we design an alternative leaderboard mechanism that satisfies local stability while eliciting only $\widetilde{O}(k)$ pairwise comparisons per user, where $k$ is the size of the prefix for which stability is guaranteed. Using data from LMArena, we show that standard Bradley–Terry aggregation can violate local stability in practice, whereas our method provides substantially stronger stability guarantees.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-haghtalab26a, title = {Pluralistic Leaderboards}, author = {Haghtalab, Nika and Procaccia, Ariel D. and Shao, Han and Wang, Serena Lutong and Yang, Kunhe}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {39356--39375}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/haghtalab26a/haghtalab26a.pdf}, url = {https://proceedings.mlr.press/v306/haghtalab26a.html}, abstract = {Recent leaderboard-based evaluations of large language models aggregate user feedback by fitting a Bradley–Terry model to pairwise comparisons, producing a single global ranking based on a latent quality score. While appealing for its simplicity, this approach is incompatible with heterogeneous preferences: when LLMs are used across diverse tasks and use cases, users who favor fundamentally different model behaviors can be systematically misrepresented when collapsed into a single quality score. To address this issue, we study pluralistic leaderboards that aim to remain stable with respect to heterogeneous user populations. Drawing on ideas from social choice theory, we adapt the notion of local stability, which requires that no model outside the top-$k$ positions is collectively preferred to the top-$k$ set by more than $O(1/k)$ fraction of users. Building on techniques from the social choice literature, we design an alternative leaderboard mechanism that satisfies local stability while eliciting only $\widetilde{O}(k)$ pairwise comparisons per user, where $k$ is the size of the prefix for which stability is guaranteed. Using data from LMArena, we show that standard Bradley–Terry aggregation can violate local stability in practice, whereas our method provides substantially stronger stability guarantees.} }
Endnote
%0 Conference Paper %T Pluralistic Leaderboards %A Nika Haghtalab %A Ariel D. Procaccia %A Han Shao %A Serena Lutong Wang %A Kunhe Yang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-haghtalab26a %I PMLR %P 39356--39375 %U https://proceedings.mlr.press/v306/haghtalab26a.html %V 306 %X Recent leaderboard-based evaluations of large language models aggregate user feedback by fitting a Bradley–Terry model to pairwise comparisons, producing a single global ranking based on a latent quality score. While appealing for its simplicity, this approach is incompatible with heterogeneous preferences: when LLMs are used across diverse tasks and use cases, users who favor fundamentally different model behaviors can be systematically misrepresented when collapsed into a single quality score. To address this issue, we study pluralistic leaderboards that aim to remain stable with respect to heterogeneous user populations. Drawing on ideas from social choice theory, we adapt the notion of local stability, which requires that no model outside the top-$k$ positions is collectively preferred to the top-$k$ set by more than $O(1/k)$ fraction of users. Building on techniques from the social choice literature, we design an alternative leaderboard mechanism that satisfies local stability while eliciting only $\widetilde{O}(k)$ pairwise comparisons per user, where $k$ is the size of the prefix for which stability is guaranteed. Using data from LMArena, we show that standard Bradley–Terry aggregation can violate local stability in practice, whereas our method provides substantially stronger stability guarantees.
APA
Haghtalab, N., Procaccia, A.D., Shao, H., Wang, S.L. & Yang, K.. (2026). Pluralistic Leaderboards. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:39356-39375 Available from https://proceedings.mlr.press/v306/haghtalab26a.html.

Related Material