Good Initializations of Variational Bayes for Deep Models

Simone Rossi, Pietro Michiardi, Maurizio Filippone
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:5487-5497, 2019.

Abstract

Stochastic variational inference is an established way to carry out approximate Bayesian inference for deep models flexibly and at scale. While there have been effective proposals for good initializations for loss minimization in deep learning, far less attention has been devoted to the issue of initialization of stochastic variational inference. We address this by proposing a novel layer-wise initialization strategy based on Bayesian linear models. The proposed method is extensively validated on regression and classification tasks, including Bayesian Deep Nets and Conv Nets, showing faster and better convergence compared to alternatives inspired by the literature on initializations for loss minimization.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-rossi19a, title = {Good Initializations of Variational {B}ayes for Deep Models}, author = {Rossi, Simone and Michiardi, Pietro and Filippone, Maurizio}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {5487--5497}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/rossi19a/rossi19a.pdf}, url = {https://proceedings.mlr.press/v97/rossi19a.html}, abstract = {Stochastic variational inference is an established way to carry out approximate Bayesian inference for deep models flexibly and at scale. While there have been effective proposals for good initializations for loss minimization in deep learning, far less attention has been devoted to the issue of initialization of stochastic variational inference. We address this by proposing a novel layer-wise initialization strategy based on Bayesian linear models. The proposed method is extensively validated on regression and classification tasks, including Bayesian Deep Nets and Conv Nets, showing faster and better convergence compared to alternatives inspired by the literature on initializations for loss minimization.} }
Endnote
%0 Conference Paper %T Good Initializations of Variational Bayes for Deep Models %A Simone Rossi %A Pietro Michiardi %A Maurizio Filippone %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-rossi19a %I PMLR %P 5487--5497 %U https://proceedings.mlr.press/v97/rossi19a.html %V 97 %X Stochastic variational inference is an established way to carry out approximate Bayesian inference for deep models flexibly and at scale. While there have been effective proposals for good initializations for loss minimization in deep learning, far less attention has been devoted to the issue of initialization of stochastic variational inference. We address this by proposing a novel layer-wise initialization strategy based on Bayesian linear models. The proposed method is extensively validated on regression and classification tasks, including Bayesian Deep Nets and Conv Nets, showing faster and better convergence compared to alternatives inspired by the literature on initializations for loss minimization.
APA
Rossi, S., Michiardi, P. & Filippone, M.. (2019). Good Initializations of Variational Bayes for Deep Models. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:5487-5497 Available from https://proceedings.mlr.press/v97/rossi19a.html.

Related Material