On the Convergence Rate of Gaussianization with Random Rotations

Felix Draxler, Lars Kühmichel, Armand Rousselot, Jens Müller, Christoph Schnoerr, Ullrich Koethe
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:8449-8468, 2023.

Abstract

Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model is unable to capture dependencies between dimensions. Empirically, we find the same linear increase in cost for arbitrary input $p(x)$, but observe favorable scaling for some distributions. We explore potential speed-ups and formulate challenges for further research.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-draxler23a, title = {On the Convergence Rate of Gaussianization with Random Rotations}, author = {Draxler, Felix and K\"{u}hmichel, Lars and Rousselot, Armand and M\"{u}ller, Jens and Schnoerr, Christoph and Koethe, Ullrich}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {8449--8468}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/draxler23a/draxler23a.pdf}, url = {https://proceedings.mlr.press/v202/draxler23a.html}, abstract = {Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model is unable to capture dependencies between dimensions. Empirically, we find the same linear increase in cost for arbitrary input $p(x)$, but observe favorable scaling for some distributions. We explore potential speed-ups and formulate challenges for further research.} }
Endnote
%0 Conference Paper %T On the Convergence Rate of Gaussianization with Random Rotations %A Felix Draxler %A Lars Kühmichel %A Armand Rousselot %A Jens Müller %A Christoph Schnoerr %A Ullrich Koethe %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-draxler23a %I PMLR %P 8449--8468 %U https://proceedings.mlr.press/v202/draxler23a.html %V 202 %X Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model is unable to capture dependencies between dimensions. Empirically, we find the same linear increase in cost for arbitrary input $p(x)$, but observe favorable scaling for some distributions. We explore potential speed-ups and formulate challenges for further research.
APA
Draxler, F., Kühmichel, L., Rousselot, A., Müller, J., Schnoerr, C. & Koethe, U.. (2023). On the Convergence Rate of Gaussianization with Random Rotations. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:8449-8468 Available from https://proceedings.mlr.press/v202/draxler23a.html.

Related Material