ConjugateComputation Variational Inference : Converting Variational Inference in NonConjugate Models to Inferences in Conjugate Models
[edit]
Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:878887, 2017.
Abstract
Variational inference is computationally challenging in models that contain both conjugate and nonconjugate terms. Methods specifically designed for conjugate models, even though computationally efficient, find it difficult to deal with nonconjugate terms. On the other hand, stochasticgradient methods can handle the nonconjugate terms but they usually ignore the conjugate structure of the model which might result in slow convergence. In this paper, we propose a new algorithm called Conjugatecomputation Variational Inference (CVI) which brings the best of the two worlds together – it uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest. We derive this algorithm by using a stochastic mirrordescent method in the meanparameter space, and then expressing each gradient step as a variational inference in a conjugate model. We demonstrate our algorithm’s applicability to a large class of models and establish its convergence. Our experimental results show that our method converges much faster than the methods that ignore the conjugate structure of the model.
Related Material


