Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures

Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, Romain Couillet
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:8573-8582, 2020.

Abstract

This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called \emph{concentrated} random vectors. Further exploiting the fact that Gram matrices, of the type $G = X^\intercal X$ with $X=[x_1,\ldots,x_n]\in \mathbb{R}^{p\times n}$ and $x_i$ independent concentrated random vectors from a mixture model, behave asymptotically (as $n,p\to \infty$) as if the $x_i$ were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-seddik20a, title = {Random Matrix Theory Proves that Deep Learning Representations of {GAN}-data Behave as {G}aussian Mixtures}, author = {Seddik, Mohamed El Amine and Louart, Cosme and Tamaazousti, Mohamed and Couillet, Romain}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {8573--8582}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/seddik20a/seddik20a.pdf}, url = {https://proceedings.mlr.press/v119/seddik20a.html}, abstract = {This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called \emph{concentrated} random vectors. Further exploiting the fact that Gram matrices, of the type $G = X^\intercal X$ with $X=[x_1,\ldots,x_n]\in \mathbb{R}^{p\times n}$ and $x_i$ independent concentrated random vectors from a mixture model, behave asymptotically (as $n,p\to \infty$) as if the $x_i$ were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.} }
Endnote
%0 Conference Paper %T Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures %A Mohamed El Amine Seddik %A Cosme Louart %A Mohamed Tamaazousti %A Romain Couillet %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-seddik20a %I PMLR %P 8573--8582 %U https://proceedings.mlr.press/v119/seddik20a.html %V 119 %X This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called \emph{concentrated} random vectors. Further exploiting the fact that Gram matrices, of the type $G = X^\intercal X$ with $X=[x_1,\ldots,x_n]\in \mathbb{R}^{p\times n}$ and $x_i$ independent concentrated random vectors from a mixture model, behave asymptotically (as $n,p\to \infty$) as if the $x_i$ were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.
APA
Seddik, M.E.A., Louart, C., Tamaazousti, M. & Couillet, R.. (2020). Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:8573-8582 Available from https://proceedings.mlr.press/v119/seddik20a.html.

Related Material