Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)

Yoonsoo Nam, Seok Hyeong Lee, Clémentine Carla Juliette Dominé, Yeachan Park, Charles London, Wonyl Choi, Niclas Alexander Göring, Seungjai Lee
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:81897-81929, 2025.

Abstract

In physics, complex systems are often simplified into minimal, solvable models that retain only the core principles. In machine learning, layerwise linear models (e.g., linear neural networks) act as simplified representations of neural network dynamics. These models follow the dynamical feedback principle, which describes how layers mutually govern and amplify each other’s evolution. This principle extends beyond the simplified models, successfully explaining a wide range of dynamical phenomena in deep neural networks, including neural collapse, emergence, lazy and rich regimes, and grokking. In this position paper, we call for the use of layerwise linear models retaining the core principles of neural dynamical phenomena to accelerate the science of deep learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-nam25a, title = {Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena ({N}eural Collapse, Emergence, {L}azy/{R}ich Regime, and Grokking)}, author = {Nam, Yoonsoo and Lee, Seok Hyeong and Domin\'{e}, Cl\'{e}mentine Carla Juliette and Park, Yeachan and London, Charles and Choi, Wonyl and G\"{o}ring, Niclas Alexander and Lee, Seungjai}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {81897--81929}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/nam25a/nam25a.pdf}, url = {https://proceedings.mlr.press/v267/nam25a.html}, abstract = {In physics, complex systems are often simplified into minimal, solvable models that retain only the core principles. In machine learning, layerwise linear models (e.g., linear neural networks) act as simplified representations of neural network dynamics. These models follow the dynamical feedback principle, which describes how layers mutually govern and amplify each other’s evolution. This principle extends beyond the simplified models, successfully explaining a wide range of dynamical phenomena in deep neural networks, including neural collapse, emergence, lazy and rich regimes, and grokking. In this position paper, we call for the use of layerwise linear models retaining the core principles of neural dynamical phenomena to accelerate the science of deep learning.} }
Endnote
%0 Conference Paper %T Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) %A Yoonsoo Nam %A Seok Hyeong Lee %A Clémentine Carla Juliette Dominé %A Yeachan Park %A Charles London %A Wonyl Choi %A Niclas Alexander Göring %A Seungjai Lee %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-nam25a %I PMLR %P 81897--81929 %U https://proceedings.mlr.press/v267/nam25a.html %V 267 %X In physics, complex systems are often simplified into minimal, solvable models that retain only the core principles. In machine learning, layerwise linear models (e.g., linear neural networks) act as simplified representations of neural network dynamics. These models follow the dynamical feedback principle, which describes how layers mutually govern and amplify each other’s evolution. This principle extends beyond the simplified models, successfully explaining a wide range of dynamical phenomena in deep neural networks, including neural collapse, emergence, lazy and rich regimes, and grokking. In this position paper, we call for the use of layerwise linear models retaining the core principles of neural dynamical phenomena to accelerate the science of deep learning.
APA
Nam, Y., Lee, S.H., Dominé, C.C.J., Park, Y., London, C., Choi, W., Göring, N.A. & Lee, S.. (2025). Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking). Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:81897-81929 Available from https://proceedings.mlr.press/v267/nam25a.html.

Related Material