Characterizing Structural Regularities of Labeled Data in Overparameterized Models

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, Michael C Mozer
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:5034-5044, 2021.

Abstract

Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We apply the score toward understanding the dynamics of representation learning and to filter outliers during training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-jiang21k, title = {Characterizing Structural Regularities of Labeled Data in Overparameterized Models}, author = {Jiang, Ziheng and Zhang, Chiyuan and Talwar, Kunal and Mozer, Michael C}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {5034--5044}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/jiang21k/jiang21k.pdf}, url = {https://proceedings.mlr.press/v139/jiang21k.html}, abstract = {Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We apply the score toward understanding the dynamics of representation learning and to filter outliers during training.} }
Endnote
%0 Conference Paper %T Characterizing Structural Regularities of Labeled Data in Overparameterized Models %A Ziheng Jiang %A Chiyuan Zhang %A Kunal Talwar %A Michael C Mozer %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-jiang21k %I PMLR %P 5034--5044 %U https://proceedings.mlr.press/v139/jiang21k.html %V 139 %X Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We apply the score toward understanding the dynamics of representation learning and to filter outliers during training.
APA
Jiang, Z., Zhang, C., Talwar, K. & Mozer, M.C.. (2021). Characterizing Structural Regularities of Labeled Data in Overparameterized Models. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:5034-5044 Available from https://proceedings.mlr.press/v139/jiang21k.html.

Related Material