DVAE++: Discrete Variational Autoencoders with Overlapping Transformations

Arash Vahdat, William Macready, Zhengbing Bian, Amir Khoshaman, Evgeny Andriyash
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5035-5044, 2018.

Abstract

Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-vahdat18a, title = {{DVAE}++: Discrete Variational Autoencoders with Overlapping Transformations}, author = {Vahdat, Arash and Macready, William and Bian, Zhengbing and Khoshaman, Amir and Andriyash, Evgeny}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {5035--5044}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/vahdat18a/vahdat18a.pdf}, url = {https://proceedings.mlr.press/v80/vahdat18a.html}, abstract = {Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).} }
Endnote
%0 Conference Paper %T DVAE++: Discrete Variational Autoencoders with Overlapping Transformations %A Arash Vahdat %A William Macready %A Zhengbing Bian %A Amir Khoshaman %A Evgeny Andriyash %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-vahdat18a %I PMLR %P 5035--5044 %U https://proceedings.mlr.press/v80/vahdat18a.html %V 80 %X Training of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).
APA
Vahdat, A., Macready, W., Bian, Z., Khoshaman, A. & Andriyash, E.. (2018). DVAE++: Discrete Variational Autoencoders with Overlapping Transformations. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:5035-5044 Available from https://proceedings.mlr.press/v80/vahdat18a.html.

Related Material