The Variational Predictive Natural Gradient

Da Tang, Rajesh Ranganath
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6145-6154, 2019.

Abstract

Variational inference transforms posterior inference into parametric optimization thereby enabling the use of latent variable models where otherwise impractical. However, variational inference can be finicky when different variational parameters control variables that are strongly correlated under the model. Traditional natural gradients based on the variational approximation fail to correct for correlations when the approximation is not the true posterior. To address this, we construct a new natural gradient called the Variational Predictive Natural Gradient (VPNG). Unlike traditional natural gradients for variational inference, this natural gradient accounts for the relationship between model parameters and variational parameters. We demonstrate the insight with a simple example as well as the empirical value on a classification task, a deep generative model of images, and probabilistic matrix factorization for recommendation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-tang19c, title = {The Variational Predictive Natural Gradient}, author = {Tang, Da and Ranganath, Rajesh}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {6145--6154}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/tang19c/tang19c.pdf}, url = {https://proceedings.mlr.press/v97/tang19c.html}, abstract = {Variational inference transforms posterior inference into parametric optimization thereby enabling the use of latent variable models where otherwise impractical. However, variational inference can be finicky when different variational parameters control variables that are strongly correlated under the model. Traditional natural gradients based on the variational approximation fail to correct for correlations when the approximation is not the true posterior. To address this, we construct a new natural gradient called the Variational Predictive Natural Gradient (VPNG). Unlike traditional natural gradients for variational inference, this natural gradient accounts for the relationship between model parameters and variational parameters. We demonstrate the insight with a simple example as well as the empirical value on a classification task, a deep generative model of images, and probabilistic matrix factorization for recommendation.} }
Endnote
%0 Conference Paper %T The Variational Predictive Natural Gradient %A Da Tang %A Rajesh Ranganath %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-tang19c %I PMLR %P 6145--6154 %U https://proceedings.mlr.press/v97/tang19c.html %V 97 %X Variational inference transforms posterior inference into parametric optimization thereby enabling the use of latent variable models where otherwise impractical. However, variational inference can be finicky when different variational parameters control variables that are strongly correlated under the model. Traditional natural gradients based on the variational approximation fail to correct for correlations when the approximation is not the true posterior. To address this, we construct a new natural gradient called the Variational Predictive Natural Gradient (VPNG). Unlike traditional natural gradients for variational inference, this natural gradient accounts for the relationship between model parameters and variational parameters. We demonstrate the insight with a simple example as well as the empirical value on a classification task, a deep generative model of images, and probabilistic matrix factorization for recommendation.
APA
Tang, D. & Ranganath, R.. (2019). The Variational Predictive Natural Gradient. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:6145-6154 Available from https://proceedings.mlr.press/v97/tang19c.html.

Related Material