Infinitely deep neural networks as diffusion processes

Stefano Peluchetti, Stefano Favaro
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:1126-1136, 2020.

Abstract

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-peluchetti20a, title = {Infinitely deep neural networks as diffusion processes}, author = {Peluchetti, Stefano and Favaro, Stefano}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {1126--1136}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/peluchetti20a/peluchetti20a.pdf}, url = {https://proceedings.mlr.press/v108/peluchetti20a.html}, abstract = {When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.} }
Endnote
%0 Conference Paper %T Infinitely deep neural networks as diffusion processes %A Stefano Peluchetti %A Stefano Favaro %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-peluchetti20a %I PMLR %P 1126--1136 %U https://proceedings.mlr.press/v108/peluchetti20a.html %V 108 %X When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.
APA
Peluchetti, S. & Favaro, S.. (2020). Infinitely deep neural networks as diffusion processes. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:1126-1136 Available from https://proceedings.mlr.press/v108/peluchetti20a.html.

Related Material