Non-Gaussian processes and neural networks at finite widths

Sho Yaida
Proceedings of The First Mathematical and Scientific Machine Learning Conference, PMLR 107:165-192, 2020.

Abstract

Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference with weakly non-Gaussian priors.

Cite this Paper

BibTeX
@InProceedings{pmlr-v107-yaida20a, title = {Non-{G}aussian processes and neural networks at finite widths}, author = {Yaida, Sho}, booktitle = {Proceedings of The First Mathematical and Scientific Machine Learning Conference}, pages = {165--192}, year = {2020}, editor = {Lu, Jianfeng and Ward, Rachel}, volume = {107}, series = {Proceedings of Machine Learning Research}, month = {20--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v107/yaida20a/yaida20a.pdf}, url = {https://proceedings.mlr.press/v107/yaida20a.html}, abstract = {Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference with weakly non-Gaussian priors. } }
Endnote
%0 Conference Paper %T Non-Gaussian processes and neural networks at finite widths %A Sho Yaida %B Proceedings of The First Mathematical and Scientific Machine Learning Conference %C Proceedings of Machine Learning Research %D 2020 %E Jianfeng Lu %E Rachel Ward %F pmlr-v107-yaida20a %I PMLR %P 165--192 %U https://proceedings.mlr.press/v107/yaida20a.html %V 107 %X Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative procedure to perform Bayesian inference with weakly non-Gaussian priors.
APA
Yaida, S.. (2020). Non-Gaussian processes and neural networks at finite widths. Proceedings of The First Mathematical and Scientific Machine Learning Conference, in Proceedings of Machine Learning Research 107:165-192 Available from https://proceedings.mlr.press/v107/yaida20a.html.

Related Material