Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions

Mohammad Emtiyaz Khan, Reza Babanezhad Harikandeh UBC, Wu Lin, Mark Schmidt, Masashi Sugiyama
Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, PMLR R14:572-581, 2016.

Abstract

Several recent works have explored stochastic gradient methods for variational inference that exploit the geometryof the variational-parameter space. However, the theoretical properties of these methods are not well-understoodand these methods typically only apply to conditionally-conjugate models. We present a new stochastic methodfor variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-formupdates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space.Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectivesby using a more general class of divergence functions.Beyond giving a theoretical justification for a variety of recent methods, our experiments show thatnew algorithms derived in this framework lead to state of the art results on a variety of problems.Further, due to its generality, we expect that our theoretical analysis could also apply to other applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR14-khan16a, title = {Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions}, author = {Khan, Mohammad Emtiyaz and UBC, Reza Babanezhad Harikandeh and Lin, Wu and Schmidt, Mark and Sugiyama, Masashi}, booktitle = {Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence}, pages = {572--581}, year = {2016}, editor = {Ihler, Alexander and Janzing, Dominik}, volume = {R14}, series = {Proceedings of Machine Learning Research}, month = {25--29 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r14/main/assets/khan16a/khan16a.pdf}, url = {https://proceedings.mlr.press/r14/khan16a.html}, abstract = {Several recent works have explored stochastic gradient methods for variational inference that exploit the geometryof the variational-parameter space. However, the theoretical properties of these methods are not well-understoodand these methods typically only apply to conditionally-conjugate models. We present a new stochastic methodfor variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-formupdates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space.Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectivesby using a more general class of divergence functions.Beyond giving a theoretical justification for a variety of recent methods, our experiments show thatnew algorithms derived in this framework lead to state of the art results on a variety of problems.Further, due to its generality, we expect that our theoretical analysis could also apply to other applications.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions %A Mohammad Emtiyaz Khan %A Reza Babanezhad Harikandeh UBC %A Wu Lin %A Mark Schmidt %A Masashi Sugiyama %B Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2016 %E Alexander Ihler %E Dominik Janzing %F pmlr-vR14-khan16a %I PMLR %P 572--581 %U https://proceedings.mlr.press/r14/khan16a.html %V R14 %X Several recent works have explored stochastic gradient methods for variational inference that exploit the geometryof the variational-parameter space. However, the theoretical properties of these methods are not well-understoodand these methods typically only apply to conditionally-conjugate models. We present a new stochastic methodfor variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-formupdates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space.Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectivesby using a more general class of divergence functions.Beyond giving a theoretical justification for a variety of recent methods, our experiments show thatnew algorithms derived in this framework lead to state of the art results on a variety of problems.Further, due to its generality, we expect that our theoretical analysis could also apply to other applications. %Z Reissued by PMLR on 04 October 2026.
APA
Khan, M.E., UBC, R.B.H., Lin, W., Schmidt, M. & Sugiyama, M.. (2016). Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R14:572-581 Available from https://proceedings.mlr.press/r14/khan16a.html. Reissued by PMLR on 04 October 2026.

Related Material