SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services

Hongcheng Guo, Yue Wang, Shaosheng Cao, Fei Zhao, Boyang Wang, Lei Li, Liang Chen, Xinze Lyu, Zhe Xu, Yao Hu, Zhoujun Li
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:21101-21137, 2025.

Abstract

With the rapid advancement of Social Networking Services (SNS), the need for intelligent and efficient interaction within diverse platforms has become more crucial. Large Language Models (LLMs) play an important role in SNS as they possess the potential to revolutionize user experience, content generation, and communication dynamics. However, recent studies focus on isolated SNS tasks rather than a comprehensive evaluation. In this paper, we introduce SNS-Bench, specially constructed for assessing the abilities of large language models from different Social Networking Services, with a wide range of SNS-related information. SNS-Bench encompasses 8 different tasks such as note classification, query content relevance, and highlight words generation in comments. Finally, 6,658 questions of social media text, including subjective questions, single-choice, and multiple-choice questions, are concluded in SNS-Bench. Further, we evaluate the performance of over 25+ current diverse LLMs on our SNS-Bench. Models with different sizes exhibit performance variations, yet adhere to the scaling law. Moreover, we hope provide more insights to revolutionize the techniques of social network services with LLMs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-guo25o, title = {{SNS}-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services}, author = {Guo, Hongcheng and Wang, Yue and Cao, Shaosheng and Zhao, Fei and Wang, Boyang and Li, Lei and Chen, Liang and Lyu, Xinze and Xu, Zhe and Hu, Yao and Li, Zhoujun}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {21101--21137}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/guo25o/guo25o.pdf}, url = {https://proceedings.mlr.press/v267/guo25o.html}, abstract = {With the rapid advancement of Social Networking Services (SNS), the need for intelligent and efficient interaction within diverse platforms has become more crucial. Large Language Models (LLMs) play an important role in SNS as they possess the potential to revolutionize user experience, content generation, and communication dynamics. However, recent studies focus on isolated SNS tasks rather than a comprehensive evaluation. In this paper, we introduce SNS-Bench, specially constructed for assessing the abilities of large language models from different Social Networking Services, with a wide range of SNS-related information. SNS-Bench encompasses 8 different tasks such as note classification, query content relevance, and highlight words generation in comments. Finally, 6,658 questions of social media text, including subjective questions, single-choice, and multiple-choice questions, are concluded in SNS-Bench. Further, we evaluate the performance of over 25+ current diverse LLMs on our SNS-Bench. Models with different sizes exhibit performance variations, yet adhere to the scaling law. Moreover, we hope provide more insights to revolutionize the techniques of social network services with LLMs.} }
Endnote
%0 Conference Paper %T SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services %A Hongcheng Guo %A Yue Wang %A Shaosheng Cao %A Fei Zhao %A Boyang Wang %A Lei Li %A Liang Chen %A Xinze Lyu %A Zhe Xu %A Yao Hu %A Zhoujun Li %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-guo25o %I PMLR %P 21101--21137 %U https://proceedings.mlr.press/v267/guo25o.html %V 267 %X With the rapid advancement of Social Networking Services (SNS), the need for intelligent and efficient interaction within diverse platforms has become more crucial. Large Language Models (LLMs) play an important role in SNS as they possess the potential to revolutionize user experience, content generation, and communication dynamics. However, recent studies focus on isolated SNS tasks rather than a comprehensive evaluation. In this paper, we introduce SNS-Bench, specially constructed for assessing the abilities of large language models from different Social Networking Services, with a wide range of SNS-related information. SNS-Bench encompasses 8 different tasks such as note classification, query content relevance, and highlight words generation in comments. Finally, 6,658 questions of social media text, including subjective questions, single-choice, and multiple-choice questions, are concluded in SNS-Bench. Further, we evaluate the performance of over 25+ current diverse LLMs on our SNS-Bench. Models with different sizes exhibit performance variations, yet adhere to the scaling law. Moreover, we hope provide more insights to revolutionize the techniques of social network services with LLMs.
APA
Guo, H., Wang, Y., Cao, S., Zhao, F., Wang, B., Li, L., Chen, L., Lyu, X., Xu, Z., Hu, Y. & Li, Z.. (2025). SNS-Bench: Defining, Building, and Assessing Capabilities of Large Language Models in Social Networking Services. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:21101-21137 Available from https://proceedings.mlr.press/v267/guo25o.html.

Related Material