Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets

Guy Hacohen, Leshem Choshen, Daphna Weinshall
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3950-3960, 2020.

Abstract

We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries – models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Specifically, when fixing the architecture, we describe synthetic datasets for which this pattern is no longer observed. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-hacohen20a, title = {Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets}, author = {Hacohen, Guy and Choshen, Leshem and Weinshall, Daphna}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {3950--3960}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/hacohen20a/hacohen20a.pdf}, url = {https://proceedings.mlr.press/v119/hacohen20a.html}, abstract = {We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries – models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Specifically, when fixing the architecture, we describe synthetic datasets for which this pattern is no longer observed. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.} }
Endnote
%0 Conference Paper %T Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets %A Guy Hacohen %A Leshem Choshen %A Daphna Weinshall %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-hacohen20a %I PMLR %P 3950--3960 %U https://proceedings.mlr.press/v119/hacohen20a.html %V 119 %X We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification benchmark. While this phenomenon is strongest for models of the same architecture, it also crosses architectural boundaries – models of different architectures start by learning the same examples, after which the more powerful model may continue to learn additional examples. We further show that this pattern of results reflects the interplay between the way neural networks learn benchmark datasets. Specifically, when fixing the architecture, we describe synthetic datasets for which this pattern is no longer observed. When fixing the dataset, we show that other learning paradigms may learn the data in a different order. We hypothesize that our results reflect how neural networks discover structure in natural datasets.
APA
Hacohen, G., Choshen, L. & Weinshall, D.. (2020). Let’s Agree to Agree: Neural Networks Share Classification Order on Real Datasets. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:3950-3960 Available from https://proceedings.mlr.press/v119/hacohen20a.html.

Related Material