Data Feedback Loops: Model-driven Amplification of Dataset Biases

Rohan Taori, Tatsunori Hashimoto
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:33883-33920, 2023.

Abstract

Datasets scraped from the internet have been critical to large-scale machine learning. Yet, its success puts the utility of future internet-derived datasets at potential risk, as model outputs begin to replace human annotations as a source of supervision. In this work, we formalize a system where interactions with one model are recorded as history and scraped as training data in the future. We then analyze its stability over time by tracking changes to a test-time bias statistic (e.g. gender bias of model predictions). We find that the degree of bias amplification is closely linked to whether the model’s outputs behave like samples from the training distribution, a behavior which we characterize and define as uniform faithfulness. Experiments in three conditional prediction scenarios – image classification, visual role-labeling, and language generation – demonstrate that models that exhibit a sampling-like behavior are more faithful and thus more stable. Based on this insight, we propose an intervention to help mitigate and stabilize unstable feedback systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-taori23a, title = {Data Feedback Loops: Model-driven Amplification of Dataset Biases}, author = {Taori, Rohan and Hashimoto, Tatsunori}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {33883--33920}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/taori23a/taori23a.pdf}, url = {https://proceedings.mlr.press/v202/taori23a.html}, abstract = {Datasets scraped from the internet have been critical to large-scale machine learning. Yet, its success puts the utility of future internet-derived datasets at potential risk, as model outputs begin to replace human annotations as a source of supervision. In this work, we formalize a system where interactions with one model are recorded as history and scraped as training data in the future. We then analyze its stability over time by tracking changes to a test-time bias statistic (e.g. gender bias of model predictions). We find that the degree of bias amplification is closely linked to whether the model’s outputs behave like samples from the training distribution, a behavior which we characterize and define as uniform faithfulness. Experiments in three conditional prediction scenarios – image classification, visual role-labeling, and language generation – demonstrate that models that exhibit a sampling-like behavior are more faithful and thus more stable. Based on this insight, we propose an intervention to help mitigate and stabilize unstable feedback systems.} }
Endnote
%0 Conference Paper %T Data Feedback Loops: Model-driven Amplification of Dataset Biases %A Rohan Taori %A Tatsunori Hashimoto %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-taori23a %I PMLR %P 33883--33920 %U https://proceedings.mlr.press/v202/taori23a.html %V 202 %X Datasets scraped from the internet have been critical to large-scale machine learning. Yet, its success puts the utility of future internet-derived datasets at potential risk, as model outputs begin to replace human annotations as a source of supervision. In this work, we formalize a system where interactions with one model are recorded as history and scraped as training data in the future. We then analyze its stability over time by tracking changes to a test-time bias statistic (e.g. gender bias of model predictions). We find that the degree of bias amplification is closely linked to whether the model’s outputs behave like samples from the training distribution, a behavior which we characterize and define as uniform faithfulness. Experiments in three conditional prediction scenarios – image classification, visual role-labeling, and language generation – demonstrate that models that exhibit a sampling-like behavior are more faithful and thus more stable. Based on this insight, we propose an intervention to help mitigate and stabilize unstable feedback systems.
APA
Taori, R. & Hashimoto, T.. (2023). Data Feedback Loops: Model-driven Amplification of Dataset Biases. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:33883-33920 Available from https://proceedings.mlr.press/v202/taori23a.html.

Related Material