Exploring the Complexity of Deep Neural Networks through Functional Equivalence

Guohao Shen
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:44643-44664, 2024.

Abstract

We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-shen24a, title = {Exploring the Complexity of Deep Neural Networks through Functional Equivalence}, author = {Shen, Guohao}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {44643--44664}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/shen24a/shen24a.pdf}, url = {https://proceedings.mlr.press/v235/shen24a.html}, abstract = {We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.} }
Endnote
%0 Conference Paper %T Exploring the Complexity of Deep Neural Networks through Functional Equivalence %A Guohao Shen %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-shen24a %I PMLR %P 44643--44664 %U https://proceedings.mlr.press/v235/shen24a.html %V 235 %X We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.
APA
Shen, G.. (2024). Exploring the Complexity of Deep Neural Networks through Functional Equivalence. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:44643-44664 Available from https://proceedings.mlr.press/v235/shen24a.html.

Related Material