Singular Value Representation: A New Graph Perspective On Neural Networks

Dan Meller, Nicolas Berkouk
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:3353-3369, 2023.

Abstract

We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon. code: a python implementation of the svr can be found at https://github.com/danmlr/svr.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-meller23a, title = {Singular Value Representation: A New Graph Perspective On Neural Networks}, author = {Meller, Dan and Berkouk, Nicolas}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {3353--3369}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/meller23a/meller23a.pdf}, url = {https://proceedings.mlr.press/v206/meller23a.html}, abstract = {We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon. code: a python implementation of the svr can be found at https://github.com/danmlr/svr.} }
Endnote
%0 Conference Paper %T Singular Value Representation: A New Graph Perspective On Neural Networks %A Dan Meller %A Nicolas Berkouk %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-meller23a %I PMLR %P 3353--3369 %U https://proceedings.mlr.press/v206/meller23a.html %V 206 %X We introduce the Singular Value Representation (SVR), a new method to represent the internal state of neural networks using SVD factorization of the weights. This construction yields a new weighted graph connecting what we call spectral neurons, that correspond to specific activation patterns of classical neurons. We derive a precise statistical framework to discriminate meaningful connections between spectral neurons for fully connected and convolutional layers. To demonstrate the usefulness of our approach for machine learning research, we highlight two discoveries we made using the SVR. First, we highlight the emergence of a dominant connection in VGG networks that spans multiple deep layers. Second, we witness, without relying on any input data, that batch normalization can induce significant connections between near-kernels of deep layers, leading to a remarkable spontaneous sparsification phenomenon. code: a python implementation of the svr can be found at https://github.com/danmlr/svr.
APA
Meller, D. & Berkouk, N.. (2023). Singular Value Representation: A New Graph Perspective On Neural Networks. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:3353-3369 Available from https://proceedings.mlr.press/v206/meller23a.html.

Related Material