The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers

Mohamed El Amine Seddik, Cosme Louart, Romain COUILLET, Mohamed Tamaazousti
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:1045-1053, 2021.

Abstract

This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-el-amine-seddik21a, title = { The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers }, author = {El Amine Seddik, Mohamed and Louart, Cosme and COUILLET, Romain and Tamaazousti, Mohamed}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {1045--1053}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/el-amine-seddik21a/el-amine-seddik21a.pdf}, url = {https://proceedings.mlr.press/v130/el-amine-seddik21a.html}, abstract = { This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs. } }
Endnote
%0 Conference Paper %T The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers %A Mohamed El Amine Seddik %A Cosme Louart %A Romain COUILLET %A Mohamed Tamaazousti %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-el-amine-seddik21a %I PMLR %P 1045--1053 %U https://proceedings.mlr.press/v130/el-amine-seddik21a.html %V 130 %X This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs.
APA
El Amine Seddik, M., Louart, C., COUILLET, R. & Tamaazousti, M.. (2021). The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:1045-1053 Available from https://proceedings.mlr.press/v130/el-amine-seddik21a.html.

Related Material