AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

Michael Hardy, Anka Reuel, Lijin Zhang, Jodi M. Casabianca, Sang T. Truong, Yash Satish Dave, Hansol Lee, Benjamin W. Domingue, Sanmi Koyejo
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:40570-40613, 2026.

Abstract

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9%$) than architecture or deployment categories in this context; (4) a manifest-score “scaling law” slope has low reliability ($R_{\beta}=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-hardy26a, title = {{AI} Cartography: Mapping the Latent Landscape of {AI} Benchmark Ecosystems}, author = {Hardy, Michael and Reuel, Anka and Zhang, Lijin and Casabianca, Jodi M. and Truong, Sang T. and Dave, Yash Satish and Lee, Hansol and Domingue, Benjamin W. and Koyejo, Sanmi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {40570--40613}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hardy26a/hardy26a.pdf}, url = {https://proceedings.mlr.press/v306/hardy26a.html}, abstract = {While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9%$) than architecture or deployment categories in this context; (4) a manifest-score “scaling law” slope has low reliability ($R_{\beta}=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.} }
Endnote
%0 Conference Paper %T AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems %A Michael Hardy %A Anka Reuel %A Lijin Zhang %A Jodi M. Casabianca %A Sang T. Truong %A Yash Satish Dave %A Hansol Lee %A Benjamin W. Domingue %A Sanmi Koyejo %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hardy26a %I PMLR %P 40570--40613 %U https://proceedings.mlr.press/v306/hardy26a.html %V 306 %X While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance ($\approx9%$) than architecture or deployment categories in this context; (4) a manifest-score “scaling law” slope has low reliability ($R_{\beta}=0.53$); by contrast, the latent general-factor size slope is highly stable across ecosystem controls ($R_g=0.97$). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.
APA
Hardy, M., Reuel, A., Zhang, L., Casabianca, J.M., Truong, S.T., Dave, Y.S., Lee, H., Domingue, B.W. & Koyejo, S.. (2026). AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:40570-40613 Available from https://proceedings.mlr.press/v306/hardy26a.html.

Related Material