A Targeted Accuracy Diagnostic for Variational Approximations

Yu Wang, Mikolaj Kasprzak, Jonathan H. Huggins
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:8351-8372, 2023.

Abstract

Variational Inference (VI) is an attractive alternative to Markov Chain Monte Carlo (MCMC) due to its computational efficiency in the case of large datasets and/or complex models with high-dimensional parameters. However, evaluating the accuracy of variational approximations remains a challenge. Existing methods characterize the quality of the whole variational distribution, which is almost always poor in realistic applications, even if specific posterior functionals such as the component-wise means or variances are accurate. Hence, these diagnostics are of practical value only in limited circumstances. To address this issue, we propose the“TArgeted Diagnostic for Distribution Approximation Accuracy” (TADDAA), which uses many short parallel MCMC chains to obtain lower bounds on the error of each posterior functional of interest. We also develop a reliability check for TADDAA to determine when the lower bounds should not be trusted. Numerical experiments validate the practical utility and computational efficiency of our approach on a range of synthetic distributions and real-data examples, including sparse logistic regression and Bayesian neural network models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-wang23l, title = {A Targeted Accuracy Diagnostic for Variational Approximations}, author = {Wang, Yu and Kasprzak, Mikolaj and Huggins, Jonathan H.}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {8351--8372}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/wang23l/wang23l.pdf}, url = {https://proceedings.mlr.press/v206/wang23l.html}, abstract = {Variational Inference (VI) is an attractive alternative to Markov Chain Monte Carlo (MCMC) due to its computational efficiency in the case of large datasets and/or complex models with high-dimensional parameters. However, evaluating the accuracy of variational approximations remains a challenge. Existing methods characterize the quality of the whole variational distribution, which is almost always poor in realistic applications, even if specific posterior functionals such as the component-wise means or variances are accurate. Hence, these diagnostics are of practical value only in limited circumstances. To address this issue, we propose the“TArgeted Diagnostic for Distribution Approximation Accuracy” (TADDAA), which uses many short parallel MCMC chains to obtain lower bounds on the error of each posterior functional of interest. We also develop a reliability check for TADDAA to determine when the lower bounds should not be trusted. Numerical experiments validate the practical utility and computational efficiency of our approach on a range of synthetic distributions and real-data examples, including sparse logistic regression and Bayesian neural network models.} }
Endnote
%0 Conference Paper %T A Targeted Accuracy Diagnostic for Variational Approximations %A Yu Wang %A Mikolaj Kasprzak %A Jonathan H. Huggins %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-wang23l %I PMLR %P 8351--8372 %U https://proceedings.mlr.press/v206/wang23l.html %V 206 %X Variational Inference (VI) is an attractive alternative to Markov Chain Monte Carlo (MCMC) due to its computational efficiency in the case of large datasets and/or complex models with high-dimensional parameters. However, evaluating the accuracy of variational approximations remains a challenge. Existing methods characterize the quality of the whole variational distribution, which is almost always poor in realistic applications, even if specific posterior functionals such as the component-wise means or variances are accurate. Hence, these diagnostics are of practical value only in limited circumstances. To address this issue, we propose the“TArgeted Diagnostic for Distribution Approximation Accuracy” (TADDAA), which uses many short parallel MCMC chains to obtain lower bounds on the error of each posterior functional of interest. We also develop a reliability check for TADDAA to determine when the lower bounds should not be trusted. Numerical experiments validate the practical utility and computational efficiency of our approach on a range of synthetic distributions and real-data examples, including sparse logistic regression and Bayesian neural network models.
APA
Wang, Y., Kasprzak, M. & Huggins, J.H.. (2023). A Targeted Accuracy Diagnostic for Variational Approximations. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:8351-8372 Available from https://proceedings.mlr.press/v206/wang23l.html.

Related Material