Automatic Differentiation Variational Inference with Mixtures

Warren Morningstar, Sharad Vikram, Cusuh Ham, Andrew Gallagher, Joshua Dillon
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:3250-3258, 2021.

Abstract

Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-morningstar21b, title = { Automatic Differentiation Variational Inference with Mixtures }, author = {Morningstar, Warren and Vikram, Sharad and Ham, Cusuh and Gallagher, Andrew and Dillon, Joshua}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {3250--3258}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/morningstar21b/morningstar21b.pdf}, url = {https://proceedings.mlr.press/v130/morningstar21b.html}, abstract = { Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data. } }
Endnote
%0 Conference Paper %T Automatic Differentiation Variational Inference with Mixtures %A Warren Morningstar %A Sharad Vikram %A Cusuh Ham %A Andrew Gallagher %A Joshua Dillon %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-morningstar21b %I PMLR %P 3250--3258 %U https://proceedings.mlr.press/v130/morningstar21b.html %V 130 %X Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data.
APA
Morningstar, W., Vikram, S., Ham, C., Gallagher, A. & Dillon, J.. (2021). Automatic Differentiation Variational Inference with Mixtures . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:3250-3258 Available from https://proceedings.mlr.press/v130/morningstar21b.html.

Related Material