A Computational Framework for Evaluating Human-likeness in LLMs’ Open-ended Human Behaviors

Yuxuan Lei, Jianxun Lian, Defu Lian, Jincenzi Wu, Tianfu Wang, Xing Xie
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:65376-65395, 2026.

Abstract

Large Language Models (LLMs) have found widespread application and research in scenarios such as role-playing and sociological simulations. Despite the growing use of LLM-based agents to simulate human activities, the extent to which their behaviors resemble human behavior remains underexplored. As diverse LLMs proliferate, the traditional Turing test is ineffective for scalable evaluation and prone to bias from human-crafted challenges, leading to unfair assessments. In this work, we propose a novel distribution-based framework that comprehensively evaluates human-likeness and believability of AI behaviors by leveraging large-scale open-ended human behavior data from web. For better evaluation, we design generic metrics to cover three principles: rationality, consistency, and diversity. Implemented across online shopping, open-topic Q&A, and urban mobility, our framework reveals that even the currently best LLM still exhibits a significant gap from real user behavior, underscoring the necessity of comprehensive research and evaluation of AI’s human-like capabilities.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-lei26c, title = {A Computational Framework for Evaluating Human-likeness in {LLM}s’ Open-ended Human Behaviors}, author = {Lei, Yuxuan and Lian, Jianxun and Lian, Defu and Wu, Jincenzi and Wang, Tianfu and Xie, Xing}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {65376--65395}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lei26c/lei26c.pdf}, url = {https://proceedings.mlr.press/v306/lei26c.html}, abstract = {Large Language Models (LLMs) have found widespread application and research in scenarios such as role-playing and sociological simulations. Despite the growing use of LLM-based agents to simulate human activities, the extent to which their behaviors resemble human behavior remains underexplored. As diverse LLMs proliferate, the traditional Turing test is ineffective for scalable evaluation and prone to bias from human-crafted challenges, leading to unfair assessments. In this work, we propose a novel distribution-based framework that comprehensively evaluates human-likeness and believability of AI behaviors by leveraging large-scale open-ended human behavior data from web. For better evaluation, we design generic metrics to cover three principles: rationality, consistency, and diversity. Implemented across online shopping, open-topic Q&A, and urban mobility, our framework reveals that even the currently best LLM still exhibits a significant gap from real user behavior, underscoring the necessity of comprehensive research and evaluation of AI’s human-like capabilities.} }
Endnote
%0 Conference Paper %T A Computational Framework for Evaluating Human-likeness in LLMs’ Open-ended Human Behaviors %A Yuxuan Lei %A Jianxun Lian %A Defu Lian %A Jincenzi Wu %A Tianfu Wang %A Xing Xie %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lei26c %I PMLR %P 65376--65395 %U https://proceedings.mlr.press/v306/lei26c.html %V 306 %X Large Language Models (LLMs) have found widespread application and research in scenarios such as role-playing and sociological simulations. Despite the growing use of LLM-based agents to simulate human activities, the extent to which their behaviors resemble human behavior remains underexplored. As diverse LLMs proliferate, the traditional Turing test is ineffective for scalable evaluation and prone to bias from human-crafted challenges, leading to unfair assessments. In this work, we propose a novel distribution-based framework that comprehensively evaluates human-likeness and believability of AI behaviors by leveraging large-scale open-ended human behavior data from web. For better evaluation, we design generic metrics to cover three principles: rationality, consistency, and diversity. Implemented across online shopping, open-topic Q&A, and urban mobility, our framework reveals that even the currently best LLM still exhibits a significant gap from real user behavior, underscoring the necessity of comprehensive research and evaluation of AI’s human-like capabilities.
APA
Lei, Y., Lian, J., Lian, D., Wu, J., Wang, T. & Xie, X.. (2026). A Computational Framework for Evaluating Human-likeness in LLMs’ Open-ended Human Behaviors. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:65376-65395 Available from https://proceedings.mlr.press/v306/lei26c.html.

Related Material