Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training

Apostolos Evangelidis, Felix Krahmer
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:28382-28410, 2026.

Abstract

A plethora of recent works has shown that for wide, overparameterized neural networks, training with Stochastic Gradient Descent (SGD) often leads to interpolation of the training data without sacrificing generalization performance. A key parameter that is not only closely connected to generalization properties, but is also closely tied to other desiderata such as robustness and resistance to adversarial perturbations is the Lipschitz constant of the neural network. While empirically, the Lipschitz constant has been shown not to increase with network width, theoretical findings only provide bounds with logarithmic growth in the width and only for the random initialization of ReLU-networks. In this work, we close this gap for neural networks with smooth activations by showing that, both at random initialization and throughout lazy training, the local Lipschitz constant of deep neural networks does not increase with network width. More precisely, we establish novel non-asymptotic (finite width) upper bounds and corroborate them by numerical experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-evangelidis26a, title = {Width Independent Bounds for the Local {L}ipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training}, author = {Evangelidis, Apostolos and Krahmer, Felix}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {28382--28410}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/evangelidis26a/evangelidis26a.pdf}, url = {https://proceedings.mlr.press/v306/evangelidis26a.html}, abstract = {A plethora of recent works has shown that for wide, overparameterized neural networks, training with Stochastic Gradient Descent (SGD) often leads to interpolation of the training data without sacrificing generalization performance. A key parameter that is not only closely connected to generalization properties, but is also closely tied to other desiderata such as robustness and resistance to adversarial perturbations is the Lipschitz constant of the neural network. While empirically, the Lipschitz constant has been shown not to increase with network width, theoretical findings only provide bounds with logarithmic growth in the width and only for the random initialization of ReLU-networks. In this work, we close this gap for neural networks with smooth activations by showing that, both at random initialization and throughout lazy training, the local Lipschitz constant of deep neural networks does not increase with network width. More precisely, we establish novel non-asymptotic (finite width) upper bounds and corroborate them by numerical experiments.} }
Endnote
%0 Conference Paper %T Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training %A Apostolos Evangelidis %A Felix Krahmer %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-evangelidis26a %I PMLR %P 28382--28410 %U https://proceedings.mlr.press/v306/evangelidis26a.html %V 306 %X A plethora of recent works has shown that for wide, overparameterized neural networks, training with Stochastic Gradient Descent (SGD) often leads to interpolation of the training data without sacrificing generalization performance. A key parameter that is not only closely connected to generalization properties, but is also closely tied to other desiderata such as robustness and resistance to adversarial perturbations is the Lipschitz constant of the neural network. While empirically, the Lipschitz constant has been shown not to increase with network width, theoretical findings only provide bounds with logarithmic growth in the width and only for the random initialization of ReLU-networks. In this work, we close this gap for neural networks with smooth activations by showing that, both at random initialization and throughout lazy training, the local Lipschitz constant of deep neural networks does not increase with network width. More precisely, we establish novel non-asymptotic (finite width) upper bounds and corroborate them by numerical experiments.
APA
Evangelidis, A. & Krahmer, F.. (2026). Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:28382-28410 Available from https://proceedings.mlr.press/v306/evangelidis26a.html.

Related Material