Curvature Meets Bispectrum: A Correspondence Theory for Transformer Gauge Invariants

Hong Wang, Kelly Wang
Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, PMLR 282:685-715, 2026.

Abstract

Understanding which parameter changes leave a Transformer’s function unchanged is essential for model comparison, optimization, and interpretability. This paper establishes a quantitative correspondence between geometric and algebraic approaches to neural network invariance, unifying two previously disconnected mathematical frameworks. We prove that Fisher-Rao curvature on the parameter-to-function quotient for multi-head attention provides a lower bound for bispectral energy in a linearized regime, revealing these two invariants as complementary aspects of the same underlying structure. Our theoretical framework yields practical benefits: a hybrid computational pipeline that substantially reduces runtime relative to pure algebraic methods while maintaining high discrimination accuracy for equivalence testing. Empirical validation across model scales from 4 to 24 heads demonstrates 98.9% validity of the theoretical bound, with the correspondence persisting through 10,000 training steps. By bridging differential geometry and harmonic analysis, we provide both theoretical insight into Transformer symmetries and efficient algorithms for identifying functionally equivalent models in practice.

Cite this Paper

BibTeX
@InProceedings{pmlr-v282-wang26c, title = {Curvature Meets Bispectrum: A Correspondence Theory for Transformer Gauge Invariants}, author = {Wang, Hong and Wang, Kelly}, booktitle = {Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations}, pages = {685--715}, year = {2026}, editor = {Acosta, Francisco and Azeglio, Simone and Tolooshams, Bahareh and van de Geijn, Chase and Shewmake, Christian and Sanborn, Sophia and Miolane, Nina}, volume = {282}, series = {Proceedings of Machine Learning Research}, month = {14 Dec 2024--07 Dec 2025}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v282/main/assets/wang26c/wang26c.pdf}, url = {https://proceedings.mlr.press/v282/wang26c.html}, abstract = {Understanding which parameter changes leave a Transformer’s function unchanged is essential for model comparison, optimization, and interpretability. This paper establishes a quantitative correspondence between geometric and algebraic approaches to neural network invariance, unifying two previously disconnected mathematical frameworks. We prove that Fisher-Rao curvature on the parameter-to-function quotient for multi-head attention provides a lower bound for bispectral energy in a linearized regime, revealing these two invariants as complementary aspects of the same underlying structure. Our theoretical framework yields practical benefits: a hybrid computational pipeline that substantially reduces runtime relative to pure algebraic methods while maintaining high discrimination accuracy for equivalence testing. Empirical validation across model scales from 4 to 24 heads demonstrates 98.9% validity of the theoretical bound, with the correspondence persisting through 10,000 training steps. By bridging differential geometry and harmonic analysis, we provide both theoretical insight into Transformer symmetries and efficient algorithms for identifying functionally equivalent models in practice.} }
Endnote
%0 Conference Paper %T Curvature Meets Bispectrum: A Correspondence Theory for Transformer Gauge Invariants %A Hong Wang %A Kelly Wang %B Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations %C Proceedings of Machine Learning Research %D 2026 %E Francisco Acosta %E Simone Azeglio %E Bahareh Tolooshams %E Chase van de Geijn %E Christian Shewmake %E Sophia Sanborn %E Nina Miolane %F pmlr-v282-wang26c %I PMLR %P 685--715 %U https://proceedings.mlr.press/v282/wang26c.html %V 282 %X Understanding which parameter changes leave a Transformer’s function unchanged is essential for model comparison, optimization, and interpretability. This paper establishes a quantitative correspondence between geometric and algebraic approaches to neural network invariance, unifying two previously disconnected mathematical frameworks. We prove that Fisher-Rao curvature on the parameter-to-function quotient for multi-head attention provides a lower bound for bispectral energy in a linearized regime, revealing these two invariants as complementary aspects of the same underlying structure. Our theoretical framework yields practical benefits: a hybrid computational pipeline that substantially reduces runtime relative to pure algebraic methods while maintaining high discrimination accuracy for equivalence testing. Empirical validation across model scales from 4 to 24 heads demonstrates 98.9% validity of the theoretical bound, with the correspondence persisting through 10,000 training steps. By bridging differential geometry and harmonic analysis, we provide both theoretical insight into Transformer symmetries and efficient algorithms for identifying functionally equivalent models in practice.
APA
Wang, H. & Wang, K.. (2026). Curvature Meets Bispectrum: A Correspondence Theory for Transformer Gauge Invariants. Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, in Proceedings of Machine Learning Research 282:685-715 Available from https://proceedings.mlr.press/v282/wang26c.html.

Related Material