Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

Wu Lin, Mohammad Emtiyaz Khan, Mark Schmidt
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:3992-4002, 2019.

Abstract

Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to minimal exponential-family (EF) approximations. In this paper, we extend their application to estimate structured approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a minimal conditional-EF representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-lin19b, title = {Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations}, author = {Lin, Wu and Khan, Mohammad Emtiyaz and Schmidt, Mark}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {3992--4002}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/lin19b/lin19b.pdf}, url = {https://proceedings.mlr.press/v97/lin19b.html}, abstract = {Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to minimal exponential-family (EF) approximations. In this paper, we extend their application to estimate structured approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a minimal conditional-EF representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.} }
Endnote
%0 Conference Paper %T Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations %A Wu Lin %A Mohammad Emtiyaz Khan %A Mark Schmidt %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-lin19b %I PMLR %P 3992--4002 %U https://proceedings.mlr.press/v97/lin19b.html %V 97 %X Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to minimal exponential-family (EF) approximations. In this paper, we extend their application to estimate structured approximations such as mixtures of EF distributions. Such approximations can fit complex, multimodal posterior distributions and are generally more accurate than unimodal EF approximations. By using a minimal conditional-EF representation of such approximations, we derive simple natural-gradient updates. Our empirical results demonstrate a faster convergence of our natural-gradient method compared to black-box gradient-based methods. Our work expands the scope of natural gradients for Bayesian inference and makes them more widely applicable than before.
APA
Lin, W., Khan, M.E. & Schmidt, M.. (2019). Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:3992-4002 Available from https://proceedings.mlr.press/v97/lin19b.html.

Related Material