Feature learning in deep classifiers through Intermediate Neural Collapse

Akshay Rangamani, Marius Lindegaard, Tomer Galanti, Tomaso A Poggio
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:28729-28745, 2023.

Abstract

In this paper, we conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories (Willshaw et. al. 1969).

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-rangamani23a, title = {Feature learning in deep classifiers through Intermediate Neural Collapse}, author = {Rangamani, Akshay and Lindegaard, Marius and Galanti, Tomer and Poggio, Tomaso A}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {28729--28745}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/rangamani23a/rangamani23a.pdf}, url = {https://proceedings.mlr.press/v202/rangamani23a.html}, abstract = {In this paper, we conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories (Willshaw et. al. 1969).} }
Endnote
%0 Conference Paper %T Feature learning in deep classifiers through Intermediate Neural Collapse %A Akshay Rangamani %A Marius Lindegaard %A Tomer Galanti %A Tomaso A Poggio %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-rangamani23a %I PMLR %P 28729--28745 %U https://proceedings.mlr.press/v202/rangamani23a.html %V 202 %X In this paper, we conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories (Willshaw et. al. 1969).
APA
Rangamani, A., Lindegaard, M., Galanti, T. & Poggio, T.A.. (2023). Feature learning in deep classifiers through Intermediate Neural Collapse. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:28729-28745 Available from https://proceedings.mlr.press/v202/rangamani23a.html.

Related Material