Nonparametric LLM Evaluation from Preference Data

Dennis Frauen, Athiya Deviyani, Mihaela Van Der Schaar, Stefan Feuerriegel
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:31575-31617, 2026.

Abstract

Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality with complex human responses such as ties. DMLRank comes with the following advantages: (i) It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows to incorporate black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-frauen26a, title = {Nonparametric {LLM} Evaluation from Preference Data}, author = {Frauen, Dennis and Deviyani, Athiya and Van Der Schaar, Mihaela and Feuerriegel, Stefan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {31575--31617}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/frauen26a/frauen26a.pdf}, url = {https://proceedings.mlr.press/v306/frauen26a.html}, abstract = {Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality with complex human responses such as ties. DMLRank comes with the following advantages: (i) It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows to incorporate black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.} }
Endnote
%0 Conference Paper %T Nonparametric LLM Evaluation from Preference Data %A Dennis Frauen %A Athiya Deviyani %A Mihaela Van Der Schaar %A Stefan Feuerriegel %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-frauen26a %I PMLR %P 31575--31617 %U https://proceedings.mlr.press/v306/frauen26a.html %V 306 %X Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards. However, many existing approaches either rely on restrictive parametric assumptions or lack valid uncertainty quantification when flexible machine learning methods are used. In this paper, we propose a nonparametric statistical framework, called DMLRank, for comparing and ranking LLMs from preference data using debiased machine learning (DML). For this, we introduce generalized average ranking scores (GARS), which generalize commonly used ranking models, including the Bradley-Terry model or PageRank/ Rank centrality with complex human responses such as ties. DMLRank comes with the following advantages: (i) It produces statistically efficient estimates of GARS ranking scores. (ii) It naturally allows to incorporate black-box machine learning methods for estimation. (iii) It can be combined with pre-trained LLM evaluators (e.g., using LLM-as-a-judge). (iv) It suggests optimal policies for collecting preference data under budget constraints. We demonstrate these advantages both theoretically and empirically using both synthetic and real-world preference datasets. In summary, our framework provides practitioners with powerful, state-of-the-art methods for comparing or ranking LLMs for leaderboards.
APA
Frauen, D., Deviyani, A., Van Der Schaar, M. & Feuerriegel, S.. (2026). Nonparametric LLM Evaluation from Preference Data. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:31575-31617 Available from https://proceedings.mlr.press/v306/frauen26a.html.

Related Material