Position: Measure Dataset Diversity, Don’t Just Claim It

Dora Zhao, Jerone Andrews, Orestis Papakyriakopoulos, Alice Xiang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:60644-60673, 2024.

Abstract

Machine learning (ML) datasets, often perceived as neutral, inherently encapsulate abstract and disputed social constructs. Dataset curators frequently employ value-laden terms such as diversity, bias, and quality to characterize datasets. Despite their prevalence, these terms lack clear definitions and validation. Our research explores the implications of this issue by analyzing "diversity" across 135 image and text datasets. Drawing from social sciences, we apply principles from measurement theory to identify considerations and offer recommendations for conceptualizing, operationalizing, and evaluating diversity in datasets. Our findings have broader implications for ML research, advocating for a more nuanced and precise approach to handling value-laden properties in dataset construction.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zhao24a, title = {Position: Measure Dataset Diversity, Don’t Just Claim It}, author = {Zhao, Dora and Andrews, Jerone and Papakyriakopoulos, Orestis and Xiang, Alice}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {60644--60673}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zhao24a/zhao24a.pdf}, url = {https://proceedings.mlr.press/v235/zhao24a.html}, abstract = {Machine learning (ML) datasets, often perceived as neutral, inherently encapsulate abstract and disputed social constructs. Dataset curators frequently employ value-laden terms such as diversity, bias, and quality to characterize datasets. Despite their prevalence, these terms lack clear definitions and validation. Our research explores the implications of this issue by analyzing "diversity" across 135 image and text datasets. Drawing from social sciences, we apply principles from measurement theory to identify considerations and offer recommendations for conceptualizing, operationalizing, and evaluating diversity in datasets. Our findings have broader implications for ML research, advocating for a more nuanced and precise approach to handling value-laden properties in dataset construction.} }
Endnote
%0 Conference Paper %T Position: Measure Dataset Diversity, Don’t Just Claim It %A Dora Zhao %A Jerone Andrews %A Orestis Papakyriakopoulos %A Alice Xiang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zhao24a %I PMLR %P 60644--60673 %U https://proceedings.mlr.press/v235/zhao24a.html %V 235 %X Machine learning (ML) datasets, often perceived as neutral, inherently encapsulate abstract and disputed social constructs. Dataset curators frequently employ value-laden terms such as diversity, bias, and quality to characterize datasets. Despite their prevalence, these terms lack clear definitions and validation. Our research explores the implications of this issue by analyzing "diversity" across 135 image and text datasets. Drawing from social sciences, we apply principles from measurement theory to identify considerations and offer recommendations for conceptualizing, operationalizing, and evaluating diversity in datasets. Our findings have broader implications for ML research, advocating for a more nuanced and precise approach to handling value-laden properties in dataset construction.
APA
Zhao, D., Andrews, J., Papakyriakopoulos, O. & Xiang, A.. (2024). Position: Measure Dataset Diversity, Don’t Just Claim It. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:60644-60673 Available from https://proceedings.mlr.press/v235/zhao24a.html.

Related Material