Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Mackenzie Puig-Hall, Narmeen Fatimah Oozeer
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:105881-105906, 2026.

Abstract

Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge’s voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-roytburg26a, title = {Are {LLM} Evaluators Really Narcissists? {S}anity Checking Self-Preference Evaluations}, author = {Roytburg, Dani and Bozoukov, Matthew and Nguyen, Matthew and Barzdukas, Jou and Puig-Hall, Mackenzie and Oozeer, Narmeen Fatimah}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {105881--105906}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/roytburg26a/roytburg26a.pdf}, url = {https://proceedings.mlr.press/v306/roytburg26a.html}, abstract = {Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge’s voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.} }
Endnote
%0 Conference Paper %T Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations %A Dani Roytburg %A Matthew Bozoukov %A Matthew Nguyen %A Jou Barzdukas %A Mackenzie Puig-Hall %A Narmeen Fatimah Oozeer %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-roytburg26a %I PMLR %P 105881--105906 %U https://proceedings.mlr.press/v306/roytburg26a.html %V 306 %X Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge’s voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.
APA
Roytburg, D., Bozoukov, M., Nguyen, M., Barzdukas, J., Puig-Hall, M. & Oozeer, N.F.. (2026). Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:105881-105906 Available from https://proceedings.mlr.press/v306/roytburg26a.html.

Related Material