Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions

Todd Huster, Jeremy Cohen, Zinan Lin, Kevin Chan, Charles Kamhoua, Nandi O. Leslie, Cho-Yu Jason Chiang, Vyas Sekar
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:4523-4532, 2021.

Abstract

Generative adversarial networks (GANs) are often billed as "universal distribution learners", but precisely what distributions they can represent and learn is still an open question. Heavy-tailed distributions are prevalent in many different domains such as financial risk-assessment, physics, and epidemiology. We observe that existing GAN architectures do a poor job of matching the asymptotic behavior of heavy-tailed distributions, a problem that we show stems from their construction. Additionally, common loss functions produce unstable or near-zero gradients when faced with the infinite moments and large distances between outlier points characteristic of heavy-tailed distributions. We address these problems with the Pareto GAN. A Pareto GAN leverages extreme value theory and the functional properties of neural networks to learn a distribution that matches the asymptotic behavior of the marginal distributions of the features. We identify issues with standard loss functions and propose the use of alternative metric spaces that enable stable and efficient learning. Finally, we evaluate our proposed approach on a variety of heavy-tailed datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-huster21a, title = {Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions}, author = {Huster, Todd and Cohen, Jeremy and Lin, Zinan and Chan, Kevin and Kamhoua, Charles and Leslie, Nandi O. and Chiang, Cho-Yu Jason and Sekar, Vyas}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {4523--4532}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/huster21a/huster21a.pdf}, url = {https://proceedings.mlr.press/v139/huster21a.html}, abstract = {Generative adversarial networks (GANs) are often billed as "universal distribution learners", but precisely what distributions they can represent and learn is still an open question. Heavy-tailed distributions are prevalent in many different domains such as financial risk-assessment, physics, and epidemiology. We observe that existing GAN architectures do a poor job of matching the asymptotic behavior of heavy-tailed distributions, a problem that we show stems from their construction. Additionally, common loss functions produce unstable or near-zero gradients when faced with the infinite moments and large distances between outlier points characteristic of heavy-tailed distributions. We address these problems with the Pareto GAN. A Pareto GAN leverages extreme value theory and the functional properties of neural networks to learn a distribution that matches the asymptotic behavior of the marginal distributions of the features. We identify issues with standard loss functions and propose the use of alternative metric spaces that enable stable and efficient learning. Finally, we evaluate our proposed approach on a variety of heavy-tailed datasets.} }
Endnote
%0 Conference Paper %T Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions %A Todd Huster %A Jeremy Cohen %A Zinan Lin %A Kevin Chan %A Charles Kamhoua %A Nandi O. Leslie %A Cho-Yu Jason Chiang %A Vyas Sekar %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-huster21a %I PMLR %P 4523--4532 %U https://proceedings.mlr.press/v139/huster21a.html %V 139 %X Generative adversarial networks (GANs) are often billed as "universal distribution learners", but precisely what distributions they can represent and learn is still an open question. Heavy-tailed distributions are prevalent in many different domains such as financial risk-assessment, physics, and epidemiology. We observe that existing GAN architectures do a poor job of matching the asymptotic behavior of heavy-tailed distributions, a problem that we show stems from their construction. Additionally, common loss functions produce unstable or near-zero gradients when faced with the infinite moments and large distances between outlier points characteristic of heavy-tailed distributions. We address these problems with the Pareto GAN. A Pareto GAN leverages extreme value theory and the functional properties of neural networks to learn a distribution that matches the asymptotic behavior of the marginal distributions of the features. We identify issues with standard loss functions and propose the use of alternative metric spaces that enable stable and efficient learning. Finally, we evaluate our proposed approach on a variety of heavy-tailed datasets.
APA
Huster, T., Cohen, J., Lin, Z., Chan, K., Kamhoua, C., Leslie, N.O., Chiang, C.J. & Sekar, V.. (2021). Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:4523-4532 Available from https://proceedings.mlr.press/v139/huster21a.html.

Related Material