Concentration of Random Feature Matrices in High-Dimensions

Zhijun Chen, Hayden Schaeffer, Rachel Ward
Proceedings of Mathematical and Scientific Machine Learning, PMLR 190:287-302, 2022.

Abstract

The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.

Cite this Paper

BibTeX
@InProceedings{pmlr-v190-chen22b, title = {Concentration of Random Feature Matrices in High-Dimensions}, author = {Chen, Zhijun and Schaeffer, Hayden and Ward, Rachel}, booktitle = {Proceedings of Mathematical and Scientific Machine Learning}, pages = {287--302}, year = {2022}, editor = {Dong, Bin and Li, Qianxiao and Wang, Lei and Xu, Zhi-Qin John}, volume = {190}, series = {Proceedings of Machine Learning Research}, month = {15--17 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v190/chen22b/chen22b.pdf}, url = {https://proceedings.mlr.press/v190/chen22b.html}, abstract = {The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.} }
Endnote
%0 Conference Paper %T Concentration of Random Feature Matrices in High-Dimensions %A Zhijun Chen %A Hayden Schaeffer %A Rachel Ward %B Proceedings of Mathematical and Scientific Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Bin Dong %E Qianxiao Li %E Lei Wang %E Zhi-Qin John Xu %F pmlr-v190-chen22b %I PMLR %P 287--302 %U https://proceedings.mlr.press/v190/chen22b.html %V 190 %X The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.
APA
Chen, Z., Schaeffer, H. & Ward, R.. (2022). Concentration of Random Feature Matrices in High-Dimensions. Proceedings of Mathematical and Scientific Machine Learning, in Proceedings of Machine Learning Research 190:287-302 Available from https://proceedings.mlr.press/v190/chen22b.html.

Related Material