Overdispersed Black-Box Variational Inference

Francisco Ruiz Columbia University, Michalis Titsias Athens University of Economics Business, david Blei Columbia and University
Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, PMLR R14:68-77, 2016.

Abstract

We introduce overdispersed black-box variational inference, a method to reduce the variance of the Monte Carlo estimator of the gradient in black-box variational inference. Instead of taking samples from the variational distribution, we use importance sampling to take samples from an overdispersed distribution in the same exponential family as the variational approximation. Our approach is general since it can be readily applied to any exponential family distribution, which is the typical choice for the variational approximation. We run experiments on two non-conjugate probabilistic models to show that our method effectively reduces the variance, and the overhead introduced by the computation of the proposal parameters and the importance weights is negligible. We find that our overdispersed importance sampling scheme provides lower variance than black-box variational inference, even when the latter uses twice the number of samples. This results in faster convergence of the black-box inference procedure.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR14-university16b, title = {Overdispersed Black-Box Variational Inference}, author = {University, Francisco Ruiz Columbia and Business, Michalis Titsias Athens University of Economics and and University, david Blei Columbia}, booktitle = {Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence}, pages = {68--77}, year = {2016}, editor = {Ihler, Alexander and Janzing, Dominik}, volume = {R14}, series = {Proceedings of Machine Learning Research}, month = {25--29 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r14/main/assets/university16b/university16b.pdf}, url = {https://proceedings.mlr.press/r14/university16b.html}, abstract = {We introduce overdispersed black-box variational inference, a method to reduce the variance of the Monte Carlo estimator of the gradient in black-box variational inference. Instead of taking samples from the variational distribution, we use importance sampling to take samples from an overdispersed distribution in the same exponential family as the variational approximation. Our approach is general since it can be readily applied to any exponential family distribution, which is the typical choice for the variational approximation. We run experiments on two non-conjugate probabilistic models to show that our method effectively reduces the variance, and the overhead introduced by the computation of the proposal parameters and the importance weights is negligible. We find that our overdispersed importance sampling scheme provides lower variance than black-box variational inference, even when the latter uses twice the number of samples. This results in faster convergence of the black-box inference procedure.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Overdispersed Black-Box Variational Inference %A Francisco Ruiz Columbia University %A Michalis Titsias Athens University of Economics Business %A david Blei Columbia and University %B Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2016 %E Alexander Ihler %E Dominik Janzing %F pmlr-vR14-university16b %I PMLR %P 68--77 %U https://proceedings.mlr.press/r14/university16b.html %V R14 %X We introduce overdispersed black-box variational inference, a method to reduce the variance of the Monte Carlo estimator of the gradient in black-box variational inference. Instead of taking samples from the variational distribution, we use importance sampling to take samples from an overdispersed distribution in the same exponential family as the variational approximation. Our approach is general since it can be readily applied to any exponential family distribution, which is the typical choice for the variational approximation. We run experiments on two non-conjugate probabilistic models to show that our method effectively reduces the variance, and the overhead introduced by the computation of the proposal parameters and the importance weights is negligible. We find that our overdispersed importance sampling scheme provides lower variance than black-box variational inference, even when the latter uses twice the number of samples. This results in faster convergence of the black-box inference procedure. %Z Reissued by PMLR on 04 October 2026.
APA
University, F.R.C., Business, M.T.A.U.o.E. & and University, d.B.C.. (2016). Overdispersed Black-Box Variational Inference. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R14:68-77 Available from https://proceedings.mlr.press/r14/university16b.html. Reissued by PMLR on 04 October 2026.

Related Material