Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization

Mohamed Chiheb Yaakoubi, Cosme Louart, Malik Tiomoko, Zhenyu Liao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:143332-143358, 2026.

Abstract

We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min–Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min–max characterization of key statistics, enabling approximation of the mean $\mu_{\hat{\theta}}$ and covariance $C_{\hat{\theta}}$ of the ERM estimator $\hat{\theta}$. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection $\hat{\theta}^\top x$ approximately follows the convolution of the (generally non-Gaussian) distribution of $\mu_{\hat{\theta}}^\top x$ with an independent centered Gaussian variable of variance $\mathrm{tr}\!\big(C_{\hat{\theta}}\,\mathbb{E}[xx^\top]\big)$. This result clarifies the scope and limits of Gaussian universality for ERMs. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yaakoubi26a, title = {Characterization of {G}aussian Universality Breakdown in High-Dimensional Empirical Risk Minimization}, author = {Yaakoubi, Mohamed Chiheb and Louart, Cosme and Tiomoko, Malik and Liao, Zhenyu}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {143332--143358}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yaakoubi26a/yaakoubi26a.pdf}, url = {https://proceedings.mlr.press/v306/yaakoubi26a.html}, abstract = {We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min–Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min–max characterization of key statistics, enabling approximation of the mean $\mu_{\hat{\theta}}$ and covariance $C_{\hat{\theta}}$ of the ERM estimator $\hat{\theta}$. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection $\hat{\theta}^\top x$ approximately follows the convolution of the (generally non-Gaussian) distribution of $\mu_{\hat{\theta}}^\top x$ with an independent centered Gaussian variable of variance $\mathrm{tr}\!\big(C_{\hat{\theta}}\,\mathbb{E}[xx^\top]\big)$. This result clarifies the scope and limits of Gaussian universality for ERMs. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.} }
Endnote
%0 Conference Paper %T Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization %A Mohamed Chiheb Yaakoubi %A Cosme Louart %A Malik Tiomoko %A Zhenyu Liao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yaakoubi26a %I PMLR %P 143332--143358 %U https://proceedings.mlr.press/v306/yaakoubi26a.html %V 306 %X We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min–Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min–max characterization of key statistics, enabling approximation of the mean $\mu_{\hat{\theta}}$ and covariance $C_{\hat{\theta}}$ of the ERM estimator $\hat{\theta}$. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection $\hat{\theta}^\top x$ approximately follows the convolution of the (generally non-Gaussian) distribution of $\mu_{\hat{\theta}}^\top x$ with an independent centered Gaussian variable of variance $\mathrm{tr}\!\big(C_{\hat{\theta}}\,\mathbb{E}[xx^\top]\big)$. This result clarifies the scope and limits of Gaussian universality for ERMs. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.
APA
Yaakoubi, M.C., Louart, C., Tiomoko, M. & Liao, Z.. (2026). Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:143332-143358 Available from https://proceedings.mlr.press/v306/yaakoubi26a.html.

Related Material