A Neural Tangent Kernel Perspective of GANs

Jean-Yves Franceschi, Emmanuel De Bézenac, Ibrahim Ayed, Mickael Chen, Sylvain Lamprier, Patrick Gallinari
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:6660-6704, 2022.

Abstract

We propose a novel theoretical framework of analysis for Generative Adversarial Networks (GANs). We reveal a fundamental flaw of previous analyses which, by incorrectly modeling GANs’ training scheme, are subject to ill-defined discriminator gradients. We overcome this issue which impedes a principled study of GAN training, solving it within our framework by taking into account the discriminator’s architecture. To this end, we leverage the theory of infinite-width neural networks for the discriminator via its Neural Tangent Kernel. We characterize the trained discriminator for a wide range of losses and establish general differentiability properties of the network. From this, we derive new insights about the convergence of the generated distribution, advancing our understanding of GANs’ training dynamics. We empirically corroborate these results via an analysis toolkit based on our framework, unveiling intuitions that are consistent with GAN practice.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-franceschi22a, title = {A Neural Tangent Kernel Perspective of {GAN}s}, author = {Franceschi, Jean-Yves and De B{\'e}zenac, Emmanuel and Ayed, Ibrahim and Chen, Mickael and Lamprier, Sylvain and Gallinari, Patrick}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {6660--6704}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/franceschi22a/franceschi22a.pdf}, url = {https://proceedings.mlr.press/v162/franceschi22a.html}, abstract = {We propose a novel theoretical framework of analysis for Generative Adversarial Networks (GANs). We reveal a fundamental flaw of previous analyses which, by incorrectly modeling GANs’ training scheme, are subject to ill-defined discriminator gradients. We overcome this issue which impedes a principled study of GAN training, solving it within our framework by taking into account the discriminator’s architecture. To this end, we leverage the theory of infinite-width neural networks for the discriminator via its Neural Tangent Kernel. We characterize the trained discriminator for a wide range of losses and establish general differentiability properties of the network. From this, we derive new insights about the convergence of the generated distribution, advancing our understanding of GANs’ training dynamics. We empirically corroborate these results via an analysis toolkit based on our framework, unveiling intuitions that are consistent with GAN practice.} }
Endnote
%0 Conference Paper %T A Neural Tangent Kernel Perspective of GANs %A Jean-Yves Franceschi %A Emmanuel De Bézenac %A Ibrahim Ayed %A Mickael Chen %A Sylvain Lamprier %A Patrick Gallinari %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-franceschi22a %I PMLR %P 6660--6704 %U https://proceedings.mlr.press/v162/franceschi22a.html %V 162 %X We propose a novel theoretical framework of analysis for Generative Adversarial Networks (GANs). We reveal a fundamental flaw of previous analyses which, by incorrectly modeling GANs’ training scheme, are subject to ill-defined discriminator gradients. We overcome this issue which impedes a principled study of GAN training, solving it within our framework by taking into account the discriminator’s architecture. To this end, we leverage the theory of infinite-width neural networks for the discriminator via its Neural Tangent Kernel. We characterize the trained discriminator for a wide range of losses and establish general differentiability properties of the network. From this, we derive new insights about the convergence of the generated distribution, advancing our understanding of GANs’ training dynamics. We empirically corroborate these results via an analysis toolkit based on our framework, unveiling intuitions that are consistent with GAN practice.
APA
Franceschi, J., De Bézenac, E., Ayed, I., Chen, M., Lamprier, S. & Gallinari, P.. (2022). A Neural Tangent Kernel Perspective of GANs. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:6660-6704 Available from https://proceedings.mlr.press/v162/franceschi22a.html.

Related Material