Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge Structures

Sougata Saha, Madhur Jindal, Saurabh Kumar Pandey, Mahardika Krisna Ihsani, Alham Fikri Aji, Monojit Choudhury
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:106891-106928, 2026.

Abstract

Most existing cultural evaluation frameworks for large language models (LLMs) focus on matching model outputs to ground-truth answers, primarily measuring factual cultural awareness. This overlooks whether models internalize broader cultural structure and pluralism. We introduce a spectral-analysis-based framework that captures large-scale macrostructural patterns in models’ cultural knowledge and evaluate eight LLMs across nine cultural domains spanning all five of Newmark’s cultural dimensions and 170 countries. Comparing with human data, we find that instruction-tuned models align more closely with human cultural structure than older models, while increased model size does not consistently improve performance. Finally, simulation-based experiments show that our proposed spectral metric better predicts a model’s ability to serve users from unfamiliar cultural backgrounds than existing ones.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-saha26a, title = {Measuring Meta-Cultural Competency: A Spectral Framework for {LLM} Knowledge Structures}, author = {Saha, Sougata and Jindal, Madhur and Pandey, Saurabh Kumar and Ihsani, Mahardika Krisna and Aji, Alham Fikri and Choudhury, Monojit}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {106891--106928}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/saha26a/saha26a.pdf}, url = {https://proceedings.mlr.press/v306/saha26a.html}, abstract = {Most existing cultural evaluation frameworks for large language models (LLMs) focus on matching model outputs to ground-truth answers, primarily measuring factual cultural awareness. This overlooks whether models internalize broader cultural structure and pluralism. We introduce a spectral-analysis-based framework that captures large-scale macrostructural patterns in models’ cultural knowledge and evaluate eight LLMs across nine cultural domains spanning all five of Newmark’s cultural dimensions and 170 countries. Comparing with human data, we find that instruction-tuned models align more closely with human cultural structure than older models, while increased model size does not consistently improve performance. Finally, simulation-based experiments show that our proposed spectral metric better predicts a model’s ability to serve users from unfamiliar cultural backgrounds than existing ones.} }
Endnote
%0 Conference Paper %T Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge Structures %A Sougata Saha %A Madhur Jindal %A Saurabh Kumar Pandey %A Mahardika Krisna Ihsani %A Alham Fikri Aji %A Monojit Choudhury %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-saha26a %I PMLR %P 106891--106928 %U https://proceedings.mlr.press/v306/saha26a.html %V 306 %X Most existing cultural evaluation frameworks for large language models (LLMs) focus on matching model outputs to ground-truth answers, primarily measuring factual cultural awareness. This overlooks whether models internalize broader cultural structure and pluralism. We introduce a spectral-analysis-based framework that captures large-scale macrostructural patterns in models’ cultural knowledge and evaluate eight LLMs across nine cultural domains spanning all five of Newmark’s cultural dimensions and 170 countries. Comparing with human data, we find that instruction-tuned models align more closely with human cultural structure than older models, while increased model size does not consistently improve performance. Finally, simulation-based experiments show that our proposed spectral metric better predicts a model’s ability to serve users from unfamiliar cultural backgrounds than existing ones.
APA
Saha, S., Jindal, M., Pandey, S.K., Ihsani, M.K., Aji, A.F. & Choudhury, M.. (2026). Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge Structures. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:106891-106928 Available from https://proceedings.mlr.press/v306/saha26a.html.

Related Material