Perceptrons and Localization of Attention’s Mean-Field Landscape

Antonio Álvarez-López, Borjan Geshkovski, Domènec Ruiz-Balet
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:2180-2206, 2026.

Abstract

The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length mean-field limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-alvarez-lopez26a, title = {Perceptrons and Localization of Attention’s Mean-Field Landscape}, author = {\'{A}lvarez-L\'{o}pez, Antonio and Geshkovski, Borjan and Ruiz-Balet, Dom\`{e}nec}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {2180--2206}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/alvarez-lopez26a/alvarez-lopez26a.pdf}, url = {https://proceedings.mlr.press/v306/alvarez-lopez26a.html}, abstract = {The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length mean-field limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.} }
Endnote
%0 Conference Paper %T Perceptrons and Localization of Attention’s Mean-Field Landscape %A Antonio Álvarez-López %A Borjan Geshkovski %A Domènec Ruiz-Balet %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-alvarez-lopez26a %I PMLR %P 2180--2206 %U https://proceedings.mlr.press/v306/alvarez-lopez26a.html %V 306 %X The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an explicit energy, and one can make sense of the infinite context length mean-field limit thanks to Wasserstein gradient flows. In this paper we study the effect of the perceptron block in this setting, and show that critical points are generically atomic and localized on subsets of the sphere.
APA
Álvarez-López, A., Geshkovski, B. & Ruiz-Balet, D.. (2026). Perceptrons and Localization of Attention’s Mean-Field Landscape. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:2180-2206 Available from https://proceedings.mlr.press/v306/alvarez-lopez26a.html.

Related Material