The effect of Leaky ReLUs on the training and generalization of overparameterized networks

Yinglong Guo, Shaohan Li, Gilad Lerman
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:4393-4401, 2024.

Abstract

We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-guo24c, title = {The effect of Leaky {ReLUs} on the training and generalization of overparameterized networks}, author = {Guo, Yinglong and Li, Shaohan and Lerman, Gilad}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {4393--4401}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/guo24c/guo24c.pdf}, url = {https://proceedings.mlr.press/v238/guo24c.html}, abstract = {We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.} }
Endnote
%0 Conference Paper %T The effect of Leaky ReLUs on the training and generalization of overparameterized networks %A Yinglong Guo %A Shaohan Li %A Gilad Lerman %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-guo24c %I PMLR %P 4393--4401 %U https://proceedings.mlr.press/v238/guo24c.html %V 238 %X We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.
APA
Guo, Y., Li, S. & Lerman, G.. (2024). The effect of Leaky ReLUs on the training and generalization of overparameterized networks. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:4393-4401 Available from https://proceedings.mlr.press/v238/guo24c.html.

Related Material