Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders

Oskar Kviman, Ricky Molén, Alexandra Hotti, Semih Kurt, Vı́ctor Elvira, Jens Lagergren
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:18008-18022, 2023.

Abstract

In this paper, we show how the mixture components cooperate when they jointly adapt to maximize the ELBO. We build upon recent advances in the multiple and adaptive importance sampling literature. We then model the mixture components using separate encoder networks and show empirically that the ELBO is monotonically non-decreasing as a function of the number of mixture components. These results hold for a range of different VAE architectures on the MNIST, FashionMNIST, and CIFAR-10 datasets. In this work, we also demonstrate that increasing the number of mixture components improves the latent-representation capabilities of the VAE on both image and single-cell datasets. This cooperative behavior motivates that using Mixture VAEs should be considered a standard approach for obtaining more flexible variational approximations. Finally, Mixture VAEs are here, for the first time, compared and combined with normalizing flows, hierarchical models and/or the VampPrior in an extensive ablation study. Multiple of our Mixture VAEs achieve state-of-the-art log-likelihood results for VAE architectures on the MNIST and FashionMNIST datasets. The experiments are reproducible using our code, provided https://github.com/Lagergren-Lab/MixtureVAEs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-kviman23a, title = {Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders}, author = {Kviman, Oskar and Mol\'{e}n, Ricky and Hotti, Alexandra and Kurt, Semih and Elvira, V\'{\i}ctor and Lagergren, Jens}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {18008--18022}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/kviman23a/kviman23a.pdf}, url = {https://proceedings.mlr.press/v202/kviman23a.html}, abstract = {In this paper, we show how the mixture components cooperate when they jointly adapt to maximize the ELBO. We build upon recent advances in the multiple and adaptive importance sampling literature. We then model the mixture components using separate encoder networks and show empirically that the ELBO is monotonically non-decreasing as a function of the number of mixture components. These results hold for a range of different VAE architectures on the MNIST, FashionMNIST, and CIFAR-10 datasets. In this work, we also demonstrate that increasing the number of mixture components improves the latent-representation capabilities of the VAE on both image and single-cell datasets. This cooperative behavior motivates that using Mixture VAEs should be considered a standard approach for obtaining more flexible variational approximations. Finally, Mixture VAEs are here, for the first time, compared and combined with normalizing flows, hierarchical models and/or the VampPrior in an extensive ablation study. Multiple of our Mixture VAEs achieve state-of-the-art log-likelihood results for VAE architectures on the MNIST and FashionMNIST datasets. The experiments are reproducible using our code, provided https://github.com/Lagergren-Lab/MixtureVAEs.} }
Endnote
%0 Conference Paper %T Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders %A Oskar Kviman %A Ricky Molén %A Alexandra Hotti %A Semih Kurt %A Vı́ctor Elvira %A Jens Lagergren %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-kviman23a %I PMLR %P 18008--18022 %U https://proceedings.mlr.press/v202/kviman23a.html %V 202 %X In this paper, we show how the mixture components cooperate when they jointly adapt to maximize the ELBO. We build upon recent advances in the multiple and adaptive importance sampling literature. We then model the mixture components using separate encoder networks and show empirically that the ELBO is monotonically non-decreasing as a function of the number of mixture components. These results hold for a range of different VAE architectures on the MNIST, FashionMNIST, and CIFAR-10 datasets. In this work, we also demonstrate that increasing the number of mixture components improves the latent-representation capabilities of the VAE on both image and single-cell datasets. This cooperative behavior motivates that using Mixture VAEs should be considered a standard approach for obtaining more flexible variational approximations. Finally, Mixture VAEs are here, for the first time, compared and combined with normalizing flows, hierarchical models and/or the VampPrior in an extensive ablation study. Multiple of our Mixture VAEs achieve state-of-the-art log-likelihood results for VAE architectures on the MNIST and FashionMNIST datasets. The experiments are reproducible using our code, provided https://github.com/Lagergren-Lab/MixtureVAEs.
APA
Kviman, O., Molén, R., Hotti, A., Kurt, S., Elvira, V. & Lagergren, J.. (2023). Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:18008-18022 Available from https://proceedings.mlr.press/v202/kviman23a.html.

Related Material