Position: Categorical Deep Learning is an Algebraic Theory of All Architectures

Bruno Gavranović, Paul Lessard, Andrew Joseph Dudzik, Tamara Von Glehn, João Guilherme Madeira Araújo, Petar Veličković
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:15209-15241, 2024.

Abstract

We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps—as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-gavranovic24a, title = {Position: Categorical Deep Learning is an Algebraic Theory of All Architectures}, author = {Gavranovi\'{c}, Bruno and Lessard, Paul and Dudzik, Andrew Joseph and Von Glehn, Tamara and Madeira Ara\'{u}jo, Jo\~{a}o Guilherme and Veli\v{c}kovi\'{c}, Petar}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {15209--15241}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/gavranovic24a/gavranovic24a.pdf}, url = {https://proceedings.mlr.press/v235/gavranovic24a.html}, abstract = {We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps—as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.} }
Endnote
%0 Conference Paper %T Position: Categorical Deep Learning is an Algebraic Theory of All Architectures %A Bruno Gavranović %A Paul Lessard %A Andrew Joseph Dudzik %A Tamara Von Glehn %A João Guilherme Madeira Araújo %A Petar Veličković %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-gavranovic24a %I PMLR %P 15209--15241 %U https://proceedings.mlr.press/v235/gavranovic24a.html %V 235 %X We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory—precisely, the universal algebra of monads valued in a 2-category of parametric maps—as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.
APA
Gavranović, B., Lessard, P., Dudzik, A.J., Von Glehn, T., Madeira Araújo, J.G. & Veličković, P.. (2024). Position: Categorical Deep Learning is an Algebraic Theory of All Architectures. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:15209-15241 Available from https://proceedings.mlr.press/v235/gavranovic24a.html.

Related Material