On Empirical Bayes Variational Autoencoder: An Excess Risk Bound

Rong Tang, Yun Yang
Proceedings of Thirty Fourth Conference on Learning Theory, PMLR 134:4068-4125, 2021.

Abstract

In this paper, we consider variational autoencoders (VAE) via empirical Bayes estimation, referred to as Empirical Bayes Variational Autoencoders (EBVAE), which is a general framework including popular VAE methods as special cases. Despite the widespread use of VAE, its theoretical aspects are less explored in the literature. Motivated by this, we establish a general theoretical framework for analyzing the excess risk associated with EBVAE under the setting of density estimation, covering both parametric and nonparametric cases, through the lens of M-estimation. As an application, we analyze the excess risk of the commonly-used EBVAE with Gaussian models and highlight the importance of covariance matrices of Gaussian encoders and decoders in obtaining a good statistical guarantee, shedding light on the empirical observations reported in the literature.

Cite this Paper

BibTeX
@InProceedings{pmlr-v134-tang21a, title = {On Empirical Bayes Variational Autoencoder: An Excess Risk Bound}, author = {Tang, Rong and Yang, Yun}, booktitle = {Proceedings of Thirty Fourth Conference on Learning Theory}, pages = {4068--4125}, year = {2021}, editor = {Belkin, Mikhail and Kpotufe, Samory}, volume = {134}, series = {Proceedings of Machine Learning Research}, month = {15--19 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v134/tang21a/tang21a.pdf}, url = {https://proceedings.mlr.press/v134/tang21a.html}, abstract = {In this paper, we consider variational autoencoders (VAE) via empirical Bayes estimation, referred to as Empirical Bayes Variational Autoencoders (EBVAE), which is a general framework including popular VAE methods as special cases. Despite the widespread use of VAE, its theoretical aspects are less explored in the literature. Motivated by this, we establish a general theoretical framework for analyzing the excess risk associated with EBVAE under the setting of density estimation, covering both parametric and nonparametric cases, through the lens of M-estimation. As an application, we analyze the excess risk of the commonly-used EBVAE with Gaussian models and highlight the importance of covariance matrices of Gaussian encoders and decoders in obtaining a good statistical guarantee, shedding light on the empirical observations reported in the literature.} }
Endnote
%0 Conference Paper %T On Empirical Bayes Variational Autoencoder: An Excess Risk Bound %A Rong Tang %A Yun Yang %B Proceedings of Thirty Fourth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2021 %E Mikhail Belkin %E Samory Kpotufe %F pmlr-v134-tang21a %I PMLR %P 4068--4125 %U https://proceedings.mlr.press/v134/tang21a.html %V 134 %X In this paper, we consider variational autoencoders (VAE) via empirical Bayes estimation, referred to as Empirical Bayes Variational Autoencoders (EBVAE), which is a general framework including popular VAE methods as special cases. Despite the widespread use of VAE, its theoretical aspects are less explored in the literature. Motivated by this, we establish a general theoretical framework for analyzing the excess risk associated with EBVAE under the setting of density estimation, covering both parametric and nonparametric cases, through the lens of M-estimation. As an application, we analyze the excess risk of the commonly-used EBVAE with Gaussian models and highlight the importance of covariance matrices of Gaussian encoders and decoders in obtaining a good statistical guarantee, shedding light on the empirical observations reported in the literature.
APA
Tang, R. & Yang, Y.. (2021). On Empirical Bayes Variational Autoencoder: An Excess Risk Bound. Proceedings of Thirty Fourth Conference on Learning Theory, in Proceedings of Machine Learning Research 134:4068-4125 Available from https://proceedings.mlr.press/v134/tang21a.html.

Related Material