Emergent Equivariance in Deep Ensembles

Jan E Gerken, Pan Kessel
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:15438-15465, 2024.

Abstract

We show that deep ensembles become equivariant for all inputs and at all training times by simply using data augmentation. Crucially, equivariance holds off-manifold and for any architecture in the infinite width limit. The equivariance is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. Neural tangent kernel theory is used to derive this result and we verify our theoretical insights using detailed numerical experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-gerken24a, title = {Emergent Equivariance in Deep Ensembles}, author = {Gerken, Jan E and Kessel, Pan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {15438--15465}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/gerken24a/gerken24a.pdf}, url = {https://proceedings.mlr.press/v235/gerken24a.html}, abstract = {We show that deep ensembles become equivariant for all inputs and at all training times by simply using data augmentation. Crucially, equivariance holds off-manifold and for any architecture in the infinite width limit. The equivariance is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. Neural tangent kernel theory is used to derive this result and we verify our theoretical insights using detailed numerical experiments.} }
Endnote
%0 Conference Paper %T Emergent Equivariance in Deep Ensembles %A Jan E Gerken %A Pan Kessel %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-gerken24a %I PMLR %P 15438--15465 %U https://proceedings.mlr.press/v235/gerken24a.html %V 235 %X We show that deep ensembles become equivariant for all inputs and at all training times by simply using data augmentation. Crucially, equivariance holds off-manifold and for any architecture in the infinite width limit. The equivariance is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. Neural tangent kernel theory is used to derive this result and we verify our theoretical insights using detailed numerical experiments.
APA
Gerken, J.E. & Kessel, P.. (2024). Emergent Equivariance in Deep Ensembles. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:15438-15465 Available from https://proceedings.mlr.press/v235/gerken24a.html.

Related Material