Inference Suboptimality in Variational Autoencoders

Chris Cremer, Xuechen Li, David Duvenaud
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1078-1086, 2018.

Abstract

Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-cremer18a, title = {Inference Suboptimality in Variational Autoencoders}, author = {Cremer, Chris and Li, Xuechen and Duvenaud, David}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1078--1086}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/cremer18a/cremer18a.pdf}, url = {https://proceedings.mlr.press/v80/cremer18a.html}, abstract = {Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.} }
Endnote
%0 Conference Paper %T Inference Suboptimality in Variational Autoencoders %A Chris Cremer %A Xuechen Li %A David Duvenaud %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-cremer18a %I PMLR %P 1078--1086 %U https://proceedings.mlr.press/v80/cremer18a.html %V 80 %X Amortized inference allows latent-variable models trained via variational learning to scale to large datasets. The quality of approximate inference is determined by two factors: a) the capacity of the variational distribution to match the true posterior and b) the ability of the recognition network to produce good variational parameters for each datapoint. We examine approximate inference in variational autoencoders in terms of these factors. We find that divergence from the true posterior is often due to imperfect recognition networks, rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.
APA
Cremer, C., Li, X. & Duvenaud, D.. (2018). Inference Suboptimality in Variational Autoencoders. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:1078-1086 Available from https://proceedings.mlr.press/v80/cremer18a.html.

Related Material