Bounding Evidence and Estimating Log-Likelihood in VAE

Łukasz Struski, Marcin Mazur, Paweł Batorski, Przemysław Spurek, Jacek Tabor
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:5036-5051, 2023.

Abstract

Many crucial problems in deep learning and statistical inference are caused by a variational gap, i.e., a difference between model evidence (log-likelihood) and evidence lower bound (ELBO). In particular, in a classical VAE setting that involves training via an ELBO cost function, it is difficult to provide a robust comparison of the effects of training between models, since we do not know a log-likelihood of data (but only its lower bound). In this paper, to deal with this problem, we introduce a general and effective upper bound, which allows us to efficiently approximate the evidence of data. We provide extensive theoretical and experimental studies of our approach, including its comparison to the other state-of-the-art upper bounds, as well as its application as a tool for the evaluation of models that were trained on various lower bounds.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-struski23a, title = {Bounding Evidence and Estimating Log-Likelihood in VAE}, author = {Struski, {\L}ukasz and Mazur, Marcin and Batorski, Pawe{\l} and Spurek, Przemys{\l}aw and Tabor, Jacek}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {5036--5051}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/struski23a/struski23a.pdf}, url = {https://proceedings.mlr.press/v206/struski23a.html}, abstract = {Many crucial problems in deep learning and statistical inference are caused by a variational gap, i.e., a difference between model evidence (log-likelihood) and evidence lower bound (ELBO). In particular, in a classical VAE setting that involves training via an ELBO cost function, it is difficult to provide a robust comparison of the effects of training between models, since we do not know a log-likelihood of data (but only its lower bound). In this paper, to deal with this problem, we introduce a general and effective upper bound, which allows us to efficiently approximate the evidence of data. We provide extensive theoretical and experimental studies of our approach, including its comparison to the other state-of-the-art upper bounds, as well as its application as a tool for the evaluation of models that were trained on various lower bounds.} }
Endnote
%0 Conference Paper %T Bounding Evidence and Estimating Log-Likelihood in VAE %A Łukasz Struski %A Marcin Mazur %A Paweł Batorski %A Przemysław Spurek %A Jacek Tabor %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-struski23a %I PMLR %P 5036--5051 %U https://proceedings.mlr.press/v206/struski23a.html %V 206 %X Many crucial problems in deep learning and statistical inference are caused by a variational gap, i.e., a difference between model evidence (log-likelihood) and evidence lower bound (ELBO). In particular, in a classical VAE setting that involves training via an ELBO cost function, it is difficult to provide a robust comparison of the effects of training between models, since we do not know a log-likelihood of data (but only its lower bound). In this paper, to deal with this problem, we introduce a general and effective upper bound, which allows us to efficiently approximate the evidence of data. We provide extensive theoretical and experimental studies of our approach, including its comparison to the other state-of-the-art upper bounds, as well as its application as a tool for the evaluation of models that were trained on various lower bounds.
APA
Struski, Ł., Mazur, M., Batorski, P., Spurek, P. & Tabor, J.. (2023). Bounding Evidence and Estimating Log-Likelihood in VAE. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:5036-5051 Available from https://proceedings.mlr.press/v206/struski23a.html.

Related Material