When LLMs Evaluate LLMs in Hiring: A Multi-Model Auditing Framework for Generative Preference Bias in AI-Assisted Recruitment

Michael Andreas Ebling, Stefania Ionescu, Nicolò Pagan
Proceedings of Fifth European Conference on Algorithmic Fairness, PMLR 350:217-222, 2026.

Abstract

Job applicants increasingly use LLMs to write cover letters, while employers use LLMs to screen them. We study whether, in this all-AI setting, the \emph{choice of model} affects hiring outcomes independently of qualifications, using a multi-agent auditing framework (8 writer models, 6 evaluator models, 10 job descriptions, 50 candidates per job). Three findings emerge: first, we identify a \emph{generative preference bias}: a consistent global ranking of writer models by evaluator acceptance, in which proprietary models are preferred and locally hosted models are systematically disadvantaged. Self-preference bias (evaluators favouring their own outputs) is an additional perturbation, strongest among the top-ranked models, though not uniformly dominant across all evaluator-condition combinations. Second, our framework identifies which evaluator models are least sensitive to the writer model used, offering actionable guidance for fairer screening design. Third, writer model choice produces measurable rank displacement: in the most adverse pairing, approximately 2 out of 25 qualified candidates are filtered out purely due to model mismatch, not because of their credentials.

Cite this Paper

BibTeX
@InProceedings{pmlr-v350-ebling26a, title = {When LLMs Evaluate LLMs in Hiring: A Multi-Model Auditing Framework for Generative Preference Bias in AI-Assisted Recruitment}, author = {Ebling, Michael Andreas and Ionescu, Stefania and Pagan, Nicol\`o}, booktitle = {Proceedings of Fifth European Conference on Algorithmic Fairness}, pages = {217--222}, year = {2026}, editor = {De Bie, Tijl and Defrance, MaryBeth and Calders, Toon and Nguyen, Dennis}, volume = {350}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v350/main/assets/ebling26a/ebling26a.pdf}, url = {https://proceedings.mlr.press/v350/ebling26a.html}, abstract = {Job applicants increasingly use LLMs to write cover letters, while employers use LLMs to screen them. We study whether, in this all-AI setting, the \emph{choice of model} affects hiring outcomes independently of qualifications, using a multi-agent auditing framework (8 writer models, 6 evaluator models, 10 job descriptions, 50 candidates per job). Three findings emerge: first, we identify a \emph{generative preference bias}: a consistent global ranking of writer models by evaluator acceptance, in which proprietary models are preferred and locally hosted models are systematically disadvantaged. Self-preference bias (evaluators favouring their own outputs) is an additional perturbation, strongest among the top-ranked models, though not uniformly dominant across all evaluator-condition combinations. Second, our framework identifies which evaluator models are least sensitive to the writer model used, offering actionable guidance for fairer screening design. Third, writer model choice produces measurable rank displacement: in the most adverse pairing, approximately 2 out of 25 qualified candidates are filtered out purely due to model mismatch, not because of their credentials.} }
Endnote
%0 Conference Paper %T When LLMs Evaluate LLMs in Hiring: A Multi-Model Auditing Framework for Generative Preference Bias in AI-Assisted Recruitment %A Michael Andreas Ebling %A Stefania Ionescu %A Nicolò Pagan %B Proceedings of Fifth European Conference on Algorithmic Fairness %C Proceedings of Machine Learning Research %D 2026 %E Tijl De Bie %E MaryBeth Defrance %E Toon Calders %E Dennis Nguyen %F pmlr-v350-ebling26a %I PMLR %P 217--222 %U https://proceedings.mlr.press/v350/ebling26a.html %V 350 %X Job applicants increasingly use LLMs to write cover letters, while employers use LLMs to screen them. We study whether, in this all-AI setting, the \emph{choice of model} affects hiring outcomes independently of qualifications, using a multi-agent auditing framework (8 writer models, 6 evaluator models, 10 job descriptions, 50 candidates per job). Three findings emerge: first, we identify a \emph{generative preference bias}: a consistent global ranking of writer models by evaluator acceptance, in which proprietary models are preferred and locally hosted models are systematically disadvantaged. Self-preference bias (evaluators favouring their own outputs) is an additional perturbation, strongest among the top-ranked models, though not uniformly dominant across all evaluator-condition combinations. Second, our framework identifies which evaluator models are least sensitive to the writer model used, offering actionable guidance for fairer screening design. Third, writer model choice produces measurable rank displacement: in the most adverse pairing, approximately 2 out of 25 qualified candidates are filtered out purely due to model mismatch, not because of their credentials.
APA
Ebling, M.A., Ionescu, S. & Pagan, N.. (2026). When LLMs Evaluate LLMs in Hiring: A Multi-Model Auditing Framework for Generative Preference Bias in AI-Assisted Recruitment. Proceedings of Fifth European Conference on Algorithmic Fairness, in Proceedings of Machine Learning Research 350:217-222 Available from https://proceedings.mlr.press/v350/ebling26a.html.

Related Material