Stable behaviour of infinitely wide deep neural networks

Stefano Peluchetti, Stefano Favaro, Sandra Fortini
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:1137-1146, 2020.

Abstract

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-peluchetti20b, title = {Stable behaviour of infinitely wide deep neural networks}, author = {Peluchetti, Stefano and Favaro, Stefano and Fortini, Sandra}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {1137--1146}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/peluchetti20b/peluchetti20b.pdf}, url = {https://proceedings.mlr.press/v108/peluchetti20b.html}, abstract = {We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.} }
Endnote
%0 Conference Paper %T Stable behaviour of infinitely wide deep neural networks %A Stefano Peluchetti %A Stefano Favaro %A Sandra Fortini %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-peluchetti20b %I PMLR %P 1137--1146 %U https://proceedings.mlr.press/v108/peluchetti20b.html %V 108 %X We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.
APA
Peluchetti, S., Favaro, S. & Fortini, S.. (2020). Stable behaviour of infinitely wide deep neural networks. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:1137-1146 Available from https://proceedings.mlr.press/v108/peluchetti20b.html.

Related Material