Information in Infinite Ensembles of Infinitely-Wide Neural Networks

[edit]

Ravid Shwartz-Ziv, Alexander A Alemi ;
Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference, PMLR 118:1-17, 2020.

Abstract

In this preliminary work, we study the generalization properties of in nite ensembles of in nitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.

Related Material