Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, Pieter Abbeel
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:2722-2730, 2019.

Abstract

Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-ho19a, title = {Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design}, author = {Ho, Jonathan and Chen, Xi and Srinivas, Aravind and Duan, Yan and Abbeel, Pieter}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {2722--2730}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/ho19a/ho19a.pdf}, url = {https://proceedings.mlr.press/v97/ho19a.html}, abstract = {Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models.} }
Endnote
%0 Conference Paper %T Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design %A Jonathan Ho %A Xi Chen %A Aravind Srinivas %A Yan Duan %A Pieter Abbeel %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-ho19a %I PMLR %P 2722--2730 %U https://proceedings.mlr.press/v97/ho19a.html %V 97 %X Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models.
APA
Ho, J., Chen, X., Srinivas, A., Duan, Y. & Abbeel, P.. (2019). Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:2722-2730 Available from https://proceedings.mlr.press/v97/ho19a.html.

Related Material