Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:10253-10300, 2026.

Abstract

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-burgin26a, title = {Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability}, author = {B\"{u}rgin, Vincent and Herbst, Daniel and Lin, Ya-Wei Eileen and Jegelka, Stefanie}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {10253--10300}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/burgin26a/burgin26a.pdf}, url = {https://proceedings.mlr.press/v306/burgin26a.html}, abstract = {Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.} }
Endnote
%0 Conference Paper %T Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability %A Vincent Bürgin %A Daniel Herbst %A Ya-Wei Eileen Lin %A Stefanie Jegelka %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-burgin26a %I PMLR %P 10253--10300 %U https://proceedings.mlr.press/v306/burgin26a.html %V 306 %X Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.
APA
Bürgin, V., Herbst, D., Lin, Y.E. & Jegelka, S.. (2026). Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:10253-10300 Available from https://proceedings.mlr.press/v306/burgin26a.html.

Related Material