Invariance of Weight Distributions in Rectified MLPs

Russell Tsuchida, Fred Roosta, Marcus Gallagher
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4995-5004, 2018.

Abstract

An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of neurons can be viewed as a mapping into a Hilbert space. We derive the equivalent kernels of MLPs with ReLU or Leaky ReLU activations for all rotationally-invariant weight distributions, generalizing a previous result that required Gaussian weight distributions. Additionally, the Central Limit Theorem is used to show that for certain activation functions, kernels corresponding to layers with weight distributions having $0$ mean and finite absolute third moment are asymptotically universal, and are well approximated by the kernel corresponding to layers with spherical Gaussian weights. In deep networks, as depth increases the equivalent kernel approaches a pathological fixed point, which can be used to argue why training randomly initialized networks can be difficult. Our results also have implications for weight initialization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-tsuchida18a, title = {Invariance of Weight Distributions in Rectified {MLP}s}, author = {Tsuchida, Russell and Roosta, Fred and Gallagher, Marcus}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {4995--5004}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/tsuchida18a/tsuchida18a.pdf}, url = {https://proceedings.mlr.press/v80/tsuchida18a.html}, abstract = {An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of neurons can be viewed as a mapping into a Hilbert space. We derive the equivalent kernels of MLPs with ReLU or Leaky ReLU activations for all rotationally-invariant weight distributions, generalizing a previous result that required Gaussian weight distributions. Additionally, the Central Limit Theorem is used to show that for certain activation functions, kernels corresponding to layers with weight distributions having $0$ mean and finite absolute third moment are asymptotically universal, and are well approximated by the kernel corresponding to layers with spherical Gaussian weights. In deep networks, as depth increases the equivalent kernel approaches a pathological fixed point, which can be used to argue why training randomly initialized networks can be difficult. Our results also have implications for weight initialization.} }
Endnote
%0 Conference Paper %T Invariance of Weight Distributions in Rectified MLPs %A Russell Tsuchida %A Fred Roosta %A Marcus Gallagher %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-tsuchida18a %I PMLR %P 4995--5004 %U https://proceedings.mlr.press/v80/tsuchida18a.html %V 80 %X An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of neurons can be viewed as a mapping into a Hilbert space. We derive the equivalent kernels of MLPs with ReLU or Leaky ReLU activations for all rotationally-invariant weight distributions, generalizing a previous result that required Gaussian weight distributions. Additionally, the Central Limit Theorem is used to show that for certain activation functions, kernels corresponding to layers with weight distributions having $0$ mean and finite absolute third moment are asymptotically universal, and are well approximated by the kernel corresponding to layers with spherical Gaussian weights. In deep networks, as depth increases the equivalent kernel approaches a pathological fixed point, which can be used to argue why training randomly initialized networks can be difficult. Our results also have implications for weight initialization.
APA
Tsuchida, R., Roosta, F. & Gallagher, M.. (2018). Invariance of Weight Distributions in Rectified MLPs. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:4995-5004 Available from https://proceedings.mlr.press/v80/tsuchida18a.html.

Related Material