The Provable Unsupervised Learning Rule Known as Batch Normalization

Rudolf H. Riedi, Randall Balestriero, Richard Baraniuk
Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026), PMLR 334(2):262-278, 2026.

Abstract

Batch normalization (BN) accelerates training and improves generalization, yet why it works remains unsettled. Prior explanations focus on the loss landscape but cannot explain why BN still helps in architectures that already optimize well. We offer a complementary geometric account: BN is an unsupervised learning rule. For networks with ReLU-like activations, BN’s centering forces every neuron’s hyperplane through the mini-batch mean, anchoring the network’s spline partition to the data, independently of the labels or loss. We prove that this anchoring makes random hyperplanes at initialization separate low-dimensional data clusters with high probability and that BN’s scaling refines partition boundaries near the data manifold. BN thus acts as a label-free mechanism for geometric adaptation and smart initialization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v334-riedi26a, title = {The Provable Unsupervised Learning Rule Known as Batch Normalization}, author = {Riedi, Rudolf H. and Balestriero, Randall and Baraniuk, Richard}, booktitle = {Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026)}, pages = {262--278}, year = {2026}, editor = {Berman, Eddie and Bernárdez, Guillermo and Chen, Samantha and Cloninger, Alex and Doster, Timothy and Emerson, Tegan and Grigsby, J. Elisenda and Kvinge, Henry and Lawrence, Hannah and Marrinan, Tim and Myers, Audun and Papillon, Mathilde and Tahmasebi, Behrooz and Telyatnikov, Lev and Walters, Robin and Weber, Melanie and Xie, YuQing and Yeats, Eric}, volume = {334}, number = {2}, series = {Proceedings of Machine Learning Research}, month = {18--20 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v334/main/assets/riedi26a/riedi26a.pdf}, url = {https://proceedings.mlr.press/v334/riedi26a.html}, abstract = {Batch normalization (BN) accelerates training and improves generalization, yet why it works remains unsettled. Prior explanations focus on the loss landscape but cannot explain why BN still helps in architectures that already optimize well. We offer a complementary geometric account: BN is an unsupervised learning rule. For networks with ReLU-like activations, BN’s centering forces every neuron’s hyperplane through the mini-batch mean, anchoring the network’s spline partition to the data, independently of the labels or loss. We prove that this anchoring makes random hyperplanes at initialization separate low-dimensional data clusters with high probability and that BN’s scaling refines partition boundaries near the data manifold. BN thus acts as a label-free mechanism for geometric adaptation and smart initialization.} }
Endnote
%0 Conference Paper %T The Provable Unsupervised Learning Rule Known as Batch Normalization %A Rudolf H. Riedi %A Randall Balestriero %A Richard Baraniuk %B Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026) %C Proceedings of Machine Learning Research %D 2026 %E Eddie Berman %E Guillermo Bernárdez %E Samantha Chen %E Alex Cloninger %E Timothy Doster %E Tegan Emerson %E J. Elisenda Grigsby %E Henry Kvinge %E Hannah Lawrence %E Tim Marrinan %E Audun Myers %E Mathilde Papillon %E Behrooz Tahmasebi %E Lev Telyatnikov %E Robin Walters %E Melanie Weber %E YuQing Xie %E Eric Yeats %F pmlr-v334-riedi26a %I PMLR %P 262--278 %U https://proceedings.mlr.press/v334/riedi26a.html %V 334 %N 2 %X Batch normalization (BN) accelerates training and improves generalization, yet why it works remains unsettled. Prior explanations focus on the loss landscape but cannot explain why BN still helps in architectures that already optimize well. We offer a complementary geometric account: BN is an unsupervised learning rule. For networks with ReLU-like activations, BN’s centering forces every neuron’s hyperplane through the mini-batch mean, anchoring the network’s spline partition to the data, independently of the labels or loss. We prove that this anchoring makes random hyperplanes at initialization separate low-dimensional data clusters with high probability and that BN’s scaling refines partition boundaries near the data manifold. BN thus acts as a label-free mechanism for geometric adaptation and smart initialization.
APA
Riedi, R.H., Balestriero, R. & Baraniuk, R.. (2026). The Provable Unsupervised Learning Rule Known as Batch Normalization. Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026), in Proceedings of Machine Learning Research 334(2):262-278 Available from https://proceedings.mlr.press/v334/riedi26a.html.

Related Material