Boosting Variational Inference: an Optimization Perspective

Francesco Locatello, Rajiv Khanna, Joydeep Ghosh, Gunnar Ratsch
Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, PMLR 84:464-472, 2018.

Abstract

Variational inference is a popular technique to approximate a possibly intractable Bayesian posterior with a more tractable one. Recently, boosting variational inference has been proposed as a new paradigm to approximate the posterior by a mixture of densities by greedily adding components to the mixture. However, as is the case with many other variational inference algorithms, its theoretical properties have not been studied. In the present work, we study the convergence properties of this approach from a modern optimization viewpoint by establishing connections to the classic Frank-Wolfe algorithm. Our analyses yields novel theoretical insights regarding the sufficient conditions for convergence, explicit rates, and algorithmic simplifications. Since a lot of focus in previous works for variational inference has been on tractability, our work is especially important as a much needed attempt to bridge the gap between probabilistic models and their corresponding theoretical properties.

Cite this Paper

BibTeX
@InProceedings{pmlr-v84-locatello18a, title = {Boosting Variational Inference: an Optimization Perspective}, author = {Locatello, Francesco and Khanna, Rajiv and Ghosh, Joydeep and Ratsch, Gunnar}, booktitle = {Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics}, pages = {464--472}, year = {2018}, editor = {Storkey, Amos and Perez-Cruz, Fernando}, volume = {84}, series = {Proceedings of Machine Learning Research}, month = {09--11 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v84/locatello18a/locatello18a.pdf}, url = {https://proceedings.mlr.press/v84/locatello18a.html}, abstract = {Variational inference is a popular technique to approximate a possibly intractable Bayesian posterior with a more tractable one. Recently, boosting variational inference has been proposed as a new paradigm to approximate the posterior by a mixture of densities by greedily adding components to the mixture. However, as is the case with many other variational inference algorithms, its theoretical properties have not been studied. In the present work, we study the convergence properties of this approach from a modern optimization viewpoint by establishing connections to the classic Frank-Wolfe algorithm. Our analyses yields novel theoretical insights regarding the sufficient conditions for convergence, explicit rates, and algorithmic simplifications. Since a lot of focus in previous works for variational inference has been on tractability, our work is especially important as a much needed attempt to bridge the gap between probabilistic models and their corresponding theoretical properties.} }
Endnote
%0 Conference Paper %T Boosting Variational Inference: an Optimization Perspective %A Francesco Locatello %A Rajiv Khanna %A Joydeep Ghosh %A Gunnar Ratsch %B Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2018 %E Amos Storkey %E Fernando Perez-Cruz %F pmlr-v84-locatello18a %I PMLR %P 464--472 %U https://proceedings.mlr.press/v84/locatello18a.html %V 84 %X Variational inference is a popular technique to approximate a possibly intractable Bayesian posterior with a more tractable one. Recently, boosting variational inference has been proposed as a new paradigm to approximate the posterior by a mixture of densities by greedily adding components to the mixture. However, as is the case with many other variational inference algorithms, its theoretical properties have not been studied. In the present work, we study the convergence properties of this approach from a modern optimization viewpoint by establishing connections to the classic Frank-Wolfe algorithm. Our analyses yields novel theoretical insights regarding the sufficient conditions for convergence, explicit rates, and algorithmic simplifications. Since a lot of focus in previous works for variational inference has been on tractability, our work is especially important as a much needed attempt to bridge the gap between probabilistic models and their corresponding theoretical properties.
APA
Locatello, F., Khanna, R., Ghosh, J. & Ratsch, G.. (2018). Boosting Variational Inference: an Optimization Perspective. Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 84:464-472 Available from https://proceedings.mlr.press/v84/locatello18a.html.

Related Material