Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead

Tom Sühr, Florian E. Dorner, Olawale Elijah Salaudeen, Augustin Kelava, Samira Samadi
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:172133-172149, 2026.

Abstract

Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as “intelligence”, despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-spec by laying out, end-to-end, how valid measurement instruments are constructed and validated and where the ontological error enters when a human-calibrated instrument is applied to LLMs.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-suhr26a, title = {Position: Stop Evaluating {AI} with Human Tests, Develop Principled, {AI}-specific Tests instead}, author = {S\"{u}hr, Tom and Dorner, Florian E. and Salaudeen, Olawale Elijah and Kelava, Augustin and Samadi, Samira}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {172133--172149}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/suhr26a/suhr26a.pdf}, url = {https://proceedings.mlr.press/v306/suhr26a.html}, abstract = {Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as “intelligence”, despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-spec by laying out, end-to-end, how valid measurement instruments are constructed and validated and where the ontological error enters when a human-calibrated instrument is applied to LLMs.} }
Endnote
%0 Conference Paper %T Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead %A Tom Sühr %A Florian E. Dorner %A Olawale Elijah Salaudeen %A Augustin Kelava %A Samira Samadi %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-suhr26a %I PMLR %P 172133--172149 %U https://proceedings.mlr.press/v306/suhr26a.html %V 306 %X Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intelligence and personality. While these results are often interpreted as strong evidence of human-like characteristics in LLMs, this paper argues that such interpretations constitute an ontological error. Human psychological and educational tests are theory-driven measurement instruments, calibrated to a specific population. Applying these tests to non-human subjects without empirical validation, risks mischaracterizing what is being measured. Furthermore, a growing trend frames AI performance on benchmarks as measurements of traits such as “intelligence”, despite known issues with validity, data contamination, cultural bias and sensitivity to superficial prompt changes. We argue that interpreting benchmark performance as measurements of human-like traits, lacks sufficient theoretical and empirical justification. This leads to our position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. We call for the development of principled, AI-spec by laying out, end-to-end, how valid measurement instruments are constructed and validated and where the ontological error enters when a human-calibrated instrument is applied to LLMs.
APA
Sühr, T., Dorner, F.E., Salaudeen, O.E., Kelava, A. & Samadi, S.. (2026). Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:172133-172149 Available from https://proceedings.mlr.press/v306/suhr26a.html.

Related Material