Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

Kyurae Kim, Kaiwen Wu, Jisu Oh, Jacob R. Gardner
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:16853-16876, 2023.

Abstract

Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the ABC condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-kim23w, title = {Practical and Matching Gradient Variance Bounds for Black-Box Variational {B}ayesian Inference}, author = {Kim, Kyurae and Wu, Kaiwen and Oh, Jisu and Gardner, Jacob R.}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {16853--16876}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/kim23w/kim23w.pdf}, url = {https://proceedings.mlr.press/v202/kim23w.html}, abstract = {Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the ABC condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.} }
Endnote
%0 Conference Paper %T Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference %A Kyurae Kim %A Kaiwen Wu %A Jisu Oh %A Jacob R. Gardner %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-kim23w %I PMLR %P 16853--16876 %U https://proceedings.mlr.press/v202/kim23w.html %V 202 %X Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the ABC condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.
APA
Kim, K., Wu, K., Oh, J. & Gardner, J.R.. (2023). Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:16853-16876 Available from https://proceedings.mlr.press/v202/kim23w.html.

Related Material