Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks

Giovanni Luca Marchetti, Christopher J Hillar, Danica Kragic, Sophia Sanborn
Proceedings of Thirty Seventh Conference on Learning Theory, PMLR 247:3775-3797, 2024.

Abstract

In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features – a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v247-marchetti24a, title = {Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks}, author = {Marchetti, Giovanni Luca and Hillar, Christopher J and Kragic, Danica and Sanborn, Sophia}, booktitle = {Proceedings of Thirty Seventh Conference on Learning Theory}, pages = {3775--3797}, year = {2024}, editor = {Agrawal, Shipra and Roth, Aaron}, volume = {247}, series = {Proceedings of Machine Learning Research}, month = {30 Jun--03 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v247/marchetti24a/marchetti24a.pdf}, url = {https://proceedings.mlr.press/v247/marchetti24a.html}, abstract = {In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features – a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.} }
Endnote
%0 Conference Paper %T Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks %A Giovanni Luca Marchetti %A Christopher J Hillar %A Danica Kragic %A Sophia Sanborn %B Proceedings of Thirty Seventh Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2024 %E Shipra Agrawal %E Aaron Roth %F pmlr-v247-marchetti24a %I PMLR %P 3775--3797 %U https://proceedings.mlr.press/v247/marchetti24a.html %V 247 %X In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features – a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.
APA
Marchetti, G.L., Hillar, C.J., Kragic, D. & Sanborn, S.. (2024). Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks. Proceedings of Thirty Seventh Conference on Learning Theory, in Proceedings of Machine Learning Research 247:3775-3797 Available from https://proceedings.mlr.press/v247/marchetti24a.html.

Related Material