VFlow: More Expressive Generative Flows with Variational Data Augmentation

Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, Tian Tian
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1660-1669, 2020.

Abstract

Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints on generative flows. In this work, we study a previously overlooked constraint that all the intermediate representations must have the same dimensionality with the data due to invertibility, limiting the width of the network. We propose VFlow to tackle this constraint on dimensionality. VFlow augments the data with extra dimensions and defines a maximum evidence lower bound (ELBO) objective for estimating the distribution of augmented data jointly with the variational data augmentation distribution. Under mild assumptions, we show that the maximum ELBO solution of VFlow is always better than the original maximum likelihood solution. For image density modeling on the CIFAR-10 dataset, VFlow achieves a new state-of-the-art 2.98 bits per dimension.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-chen20p, title = {{VF}low: More Expressive Generative Flows with Variational Data Augmentation}, author = {Chen, Jianfei and Lu, Cheng and Chenli, Biqi and Zhu, Jun and Tian, Tian}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1660--1669}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/chen20p/chen20p.pdf}, url = {https://proceedings.mlr.press/v119/chen20p.html}, abstract = {Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints on generative flows. In this work, we study a previously overlooked constraint that all the intermediate representations must have the same dimensionality with the data due to invertibility, limiting the width of the network. We propose VFlow to tackle this constraint on dimensionality. VFlow augments the data with extra dimensions and defines a maximum evidence lower bound (ELBO) objective for estimating the distribution of augmented data jointly with the variational data augmentation distribution. Under mild assumptions, we show that the maximum ELBO solution of VFlow is always better than the original maximum likelihood solution. For image density modeling on the CIFAR-10 dataset, VFlow achieves a new state-of-the-art 2.98 bits per dimension.} }
Endnote
%0 Conference Paper %T VFlow: More Expressive Generative Flows with Variational Data Augmentation %A Jianfei Chen %A Cheng Lu %A Biqi Chenli %A Jun Zhu %A Tian Tian %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-chen20p %I PMLR %P 1660--1669 %U https://proceedings.mlr.press/v119/chen20p.html %V 119 %X Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints on generative flows. In this work, we study a previously overlooked constraint that all the intermediate representations must have the same dimensionality with the data due to invertibility, limiting the width of the network. We propose VFlow to tackle this constraint on dimensionality. VFlow augments the data with extra dimensions and defines a maximum evidence lower bound (ELBO) objective for estimating the distribution of augmented data jointly with the variational data augmentation distribution. Under mild assumptions, we show that the maximum ELBO solution of VFlow is always better than the original maximum likelihood solution. For image density modeling on the CIFAR-10 dataset, VFlow achieves a new state-of-the-art 2.98 bits per dimension.
APA
Chen, J., Lu, C., Chenli, B., Zhu, J. & Tian, T.. (2020). VFlow: More Expressive Generative Flows with Variational Data Augmentation. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1660-1669 Available from https://proceedings.mlr.press/v119/chen20p.html.

Related Material