On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows

Felix Draxler, Stefan Wahl, Christoph Schnoerr, Ullrich Koethe
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:11613-11641, 2024.

Abstract

We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-draxler24a, title = {On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows}, author = {Draxler, Felix and Wahl, Stefan and Schnoerr, Christoph and Koethe, Ullrich}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {11613--11641}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/draxler24a/draxler24a.pdf}, url = {https://proceedings.mlr.press/v235/draxler24a.html}, abstract = {We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.} }
Endnote
%0 Conference Paper %T On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows %A Felix Draxler %A Stefan Wahl %A Christoph Schnoerr %A Ullrich Koethe %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-draxler24a %I PMLR %P 11613--11641 %U https://proceedings.mlr.press/v235/draxler24a.html %V 235 %X We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.
APA
Draxler, F., Wahl, S., Schnoerr, C. & Koethe, U.. (2024). On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:11613-11641 Available from https://proceedings.mlr.press/v235/draxler24a.html.

Related Material