Position: AI Leaderboards Are Underserving the Global South: A Case Study from India

Sourav Banerjee, Saikat Saha
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:169332-169348, 2026.

Abstract

This position paper argues that AI leaderboards are structurally ill-suited to serving the Global South because they lack independent governance, conflict-of-interest policies, and mechanisms for metric evolution. The barrier is not missing data. High-quality regional benchmarks already exist: IndicSUPERB, MILU, and LAHAJA for India; IrokoBench for Africa; AlGhafa for Arabic. The barrier is institutional design. Global leaderboards do not include these benchmarks, and no governance mechanism compels them to do so. Commercial pressure corrects leaderboard failures when paying customers in the Global North are affected. The Global South lacks equivalent leverage. Without governance, failures affecting Hindi, Swahili, or Arabic speakers persist indefinitely as documented but unaddressed gaps. Using India as a case study (1.4 billion people, 22 scheduled languages, high-quality benchmarks but no trusted aggregation), we report findings from a consultation with 82 AI practitioners showing strong preferences for formal, non-government governance and for disclosure-based conflict management. Our contribution is institutional, not technical: we argue that and why governance is the load-bearing solution, with India as the worked case.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-banerjee26c, title = {Position: {AI} Leaderboards Are Underserving the Global South: A Case Study from India}, author = {Banerjee, Sourav and Saha, Saikat}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {169332--169348}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/banerjee26c/banerjee26c.pdf}, url = {https://proceedings.mlr.press/v306/banerjee26c.html}, abstract = {This position paper argues that AI leaderboards are structurally ill-suited to serving the Global South because they lack independent governance, conflict-of-interest policies, and mechanisms for metric evolution. The barrier is not missing data. High-quality regional benchmarks already exist: IndicSUPERB, MILU, and LAHAJA for India; IrokoBench for Africa; AlGhafa for Arabic. The barrier is institutional design. Global leaderboards do not include these benchmarks, and no governance mechanism compels them to do so. Commercial pressure corrects leaderboard failures when paying customers in the Global North are affected. The Global South lacks equivalent leverage. Without governance, failures affecting Hindi, Swahili, or Arabic speakers persist indefinitely as documented but unaddressed gaps. Using India as a case study (1.4 billion people, 22 scheduled languages, high-quality benchmarks but no trusted aggregation), we report findings from a consultation with 82 AI practitioners showing strong preferences for formal, non-government governance and for disclosure-based conflict management. Our contribution is institutional, not technical: we argue that and why governance is the load-bearing solution, with India as the worked case.} }
Endnote
%0 Conference Paper %T Position: AI Leaderboards Are Underserving the Global South: A Case Study from India %A Sourav Banerjee %A Saikat Saha %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-banerjee26c %I PMLR %P 169332--169348 %U https://proceedings.mlr.press/v306/banerjee26c.html %V 306 %X This position paper argues that AI leaderboards are structurally ill-suited to serving the Global South because they lack independent governance, conflict-of-interest policies, and mechanisms for metric evolution. The barrier is not missing data. High-quality regional benchmarks already exist: IndicSUPERB, MILU, and LAHAJA for India; IrokoBench for Africa; AlGhafa for Arabic. The barrier is institutional design. Global leaderboards do not include these benchmarks, and no governance mechanism compels them to do so. Commercial pressure corrects leaderboard failures when paying customers in the Global North are affected. The Global South lacks equivalent leverage. Without governance, failures affecting Hindi, Swahili, or Arabic speakers persist indefinitely as documented but unaddressed gaps. Using India as a case study (1.4 billion people, 22 scheduled languages, high-quality benchmarks but no trusted aggregation), we report findings from a consultation with 82 AI practitioners showing strong preferences for formal, non-government governance and for disclosure-based conflict management. Our contribution is institutional, not technical: we argue that and why governance is the load-bearing solution, with India as the worked case.
APA
Banerjee, S. & Saha, S.. (2026). Position: AI Leaderboards Are Underserving the Global South: A Case Study from India. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:169332-169348 Available from https://proceedings.mlr.press/v306/banerjee26c.html.

Related Material