Position: Stop Using Culturally Biased Human Cognitive Benchmarks to Evaluate LLMs

Carla R. Troper
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:172330-172336, 2026.

Abstract

Recent work uses human cognitive benchmarks to evaluate how LLMs represent concepts, claiming to assess "human-like" understanding. This position paper argues that this approach is misguided: these benchmarks come from narrow, typically Western populations yet are treated as universal standards, despite cross-cultural research showing culture shapes how people think, not just what they think about. LLMs trained on global multilingual data should not be expected to mirror thinking patterns from limited groups. Moreover, LLM outputs can shift with minor changes in prompting, unlike the stable human mental structures these benchmarks were designed to measure. These problems show up as contradictory findings across studies, making benchmark results poor evidence for claims about how LLMs represent concepts. We call for evaluation approaches designed for what LLMs actually are—systems trained on diverse global data—rather than tests measuring how closely they match a single population’s way of thinking.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-troper26a, title = {Position: Stop Using Culturally Biased Human Cognitive Benchmarks to Evaluate {LLM}s}, author = {Troper, Carla R.}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {172330--172336}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/troper26a/troper26a.pdf}, url = {https://proceedings.mlr.press/v306/troper26a.html}, abstract = {Recent work uses human cognitive benchmarks to evaluate how LLMs represent concepts, claiming to assess "human-like" understanding. This position paper argues that this approach is misguided: these benchmarks come from narrow, typically Western populations yet are treated as universal standards, despite cross-cultural research showing culture shapes how people think, not just what they think about. LLMs trained on global multilingual data should not be expected to mirror thinking patterns from limited groups. Moreover, LLM outputs can shift with minor changes in prompting, unlike the stable human mental structures these benchmarks were designed to measure. These problems show up as contradictory findings across studies, making benchmark results poor evidence for claims about how LLMs represent concepts. We call for evaluation approaches designed for what LLMs actually are—systems trained on diverse global data—rather than tests measuring how closely they match a single population’s way of thinking.} }
Endnote
%0 Conference Paper %T Position: Stop Using Culturally Biased Human Cognitive Benchmarks to Evaluate LLMs %A Carla R. Troper %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-troper26a %I PMLR %P 172330--172336 %U https://proceedings.mlr.press/v306/troper26a.html %V 306 %X Recent work uses human cognitive benchmarks to evaluate how LLMs represent concepts, claiming to assess "human-like" understanding. This position paper argues that this approach is misguided: these benchmarks come from narrow, typically Western populations yet are treated as universal standards, despite cross-cultural research showing culture shapes how people think, not just what they think about. LLMs trained on global multilingual data should not be expected to mirror thinking patterns from limited groups. Moreover, LLM outputs can shift with minor changes in prompting, unlike the stable human mental structures these benchmarks were designed to measure. These problems show up as contradictory findings across studies, making benchmark results poor evidence for claims about how LLMs represent concepts. We call for evaluation approaches designed for what LLMs actually are—systems trained on diverse global data—rather than tests measuring how closely they match a single population’s way of thinking.
APA
Troper, C.R.. (2026). Position: Stop Using Culturally Biased Human Cognitive Benchmarks to Evaluate LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:172330-172336 Available from https://proceedings.mlr.press/v306/troper26a.html.

Related Material