Precise Asymptotic Analysis of Deep Random Feature Models

David Bosch, Ashkan Panahi, Babak Hassibi
Proceedings of Thirty Sixth Conference on Learning Theory, PMLR 195:4132-4179, 2023.

Abstract

We provide exact asymptotic expressions for the performance of regression by an $L-$layer deep random feature (RF) model, where the input is mapped through multiple random embedding and non-linear activation functions. For this purpose, we establish two key steps: First, we prove a novel universality result for RF models and deterministic data, by which we demonstrate that a deep random feature model is equivalent to a deep linear Gaussian model that matches it in the first and second moments, at each layer. Second, we make use of the convex Gaussian Min-Max theorem multiple times to obtain the exact behavior of deep RF models. We further characterize the variation of the eigendistribution in different layers of the equivalent Gaussian model, demonstrating that depth has a tangible effect on model performance despite the fact that only the last layer of the model is being trained.

Cite this Paper

BibTeX
@InProceedings{pmlr-v195-bosch23a, title = {Precise Asymptotic Analysis of Deep Random Feature Models}, author = {Bosch, David and Panahi, Ashkan and Hassibi, Babak}, booktitle = {Proceedings of Thirty Sixth Conference on Learning Theory}, pages = {4132--4179}, year = {2023}, editor = {Neu, Gergely and Rosasco, Lorenzo}, volume = {195}, series = {Proceedings of Machine Learning Research}, month = {12--15 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v195/bosch23a/bosch23a.pdf}, url = {https://proceedings.mlr.press/v195/bosch23a.html}, abstract = {We provide exact asymptotic expressions for the performance of regression by an $L-$layer deep random feature (RF) model, where the input is mapped through multiple random embedding and non-linear activation functions. For this purpose, we establish two key steps: First, we prove a novel universality result for RF models and deterministic data, by which we demonstrate that a deep random feature model is equivalent to a deep linear Gaussian model that matches it in the first and second moments, at each layer. Second, we make use of the convex Gaussian Min-Max theorem multiple times to obtain the exact behavior of deep RF models. We further characterize the variation of the eigendistribution in different layers of the equivalent Gaussian model, demonstrating that depth has a tangible effect on model performance despite the fact that only the last layer of the model is being trained. } }
Endnote
%0 Conference Paper %T Precise Asymptotic Analysis of Deep Random Feature Models %A David Bosch %A Ashkan Panahi %A Babak Hassibi %B Proceedings of Thirty Sixth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2023 %E Gergely Neu %E Lorenzo Rosasco %F pmlr-v195-bosch23a %I PMLR %P 4132--4179 %U https://proceedings.mlr.press/v195/bosch23a.html %V 195 %X We provide exact asymptotic expressions for the performance of regression by an $L-$layer deep random feature (RF) model, where the input is mapped through multiple random embedding and non-linear activation functions. For this purpose, we establish two key steps: First, we prove a novel universality result for RF models and deterministic data, by which we demonstrate that a deep random feature model is equivalent to a deep linear Gaussian model that matches it in the first and second moments, at each layer. Second, we make use of the convex Gaussian Min-Max theorem multiple times to obtain the exact behavior of deep RF models. We further characterize the variation of the eigendistribution in different layers of the equivalent Gaussian model, demonstrating that depth has a tangible effect on model performance despite the fact that only the last layer of the model is being trained.
APA
Bosch, D., Panahi, A. & Hassibi, B.. (2023). Precise Asymptotic Analysis of Deep Random Feature Models. Proceedings of Thirty Sixth Conference on Learning Theory, in Proceedings of Machine Learning Research 195:4132-4179 Available from https://proceedings.mlr.press/v195/bosch23a.html.

Related Material