Stochastic Neural Network with Kronecker Flow

Chin-Wei Huang, Ahmed Touati, Pascal Vincent, Gintare Karolina Dziugaite, Alexandre Lacoste, Aaron Courville
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:4184-4194, 2020.

Abstract

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and betterthan the baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-huang20a, title = {Stochastic Neural Network with Kronecker Flow}, author = {Huang, Chin-Wei and Touati, Ahmed and Vincent, Pascal and Dziugaite, Gintare Karolina and Lacoste, Alexandre and Courville, Aaron}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {4184--4194}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/huang20a/huang20a.pdf}, url = {https://proceedings.mlr.press/v108/huang20a.html}, abstract = {Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and betterthan the baselines.} }
Endnote
%0 Conference Paper %T Stochastic Neural Network with Kronecker Flow %A Chin-Wei Huang %A Ahmed Touati %A Pascal Vincent %A Gintare Karolina Dziugaite %A Alexandre Lacoste %A Aaron Courville %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-huang20a %I PMLR %P 4184--4194 %U https://proceedings.mlr.press/v108/huang20a.html %V 108 %X Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and betterthan the baselines.
APA
Huang, C., Touati, A., Vincent, P., Dziugaite, G.K., Lacoste, A. & Courville, A.. (2020). Stochastic Neural Network with Kronecker Flow. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:4184-4194 Available from https://proceedings.mlr.press/v108/huang20a.html.

Related Material