Implicit Regularization via Neural Feature Alignment

Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, Simon Lacoste-Julien
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:2269-2277, 2021.

Abstract

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment ofthe neural tangent features introduced by Jacot et al. (2018), along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths. The code for our experiments is available as https://github.com/tfjgeorge/ntk_alignment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-baratin21a, title = { Implicit Regularization via Neural Feature Alignment }, author = {Baratin, Aristide and George, Thomas and Laurent, C{\'e}sar and Devon Hjelm, R and Lajoie, Guillaume and Vincent, Pascal and Lacoste-Julien, Simon}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {2269--2277}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/baratin21a/baratin21a.pdf}, url = {https://proceedings.mlr.press/v130/baratin21a.html}, abstract = { We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment ofthe neural tangent features introduced by Jacot et al. (2018), along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths. The code for our experiments is available as https://github.com/tfjgeorge/ntk_alignment. } }
Endnote
%0 Conference Paper %T Implicit Regularization via Neural Feature Alignment %A Aristide Baratin %A Thomas George %A César Laurent %A R Devon Hjelm %A Guillaume Lajoie %A Pascal Vincent %A Simon Lacoste-Julien %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-baratin21a %I PMLR %P 2269--2277 %U https://proceedings.mlr.press/v130/baratin21a.html %V 130 %X We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment ofthe neural tangent features introduced by Jacot et al. (2018), along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths. The code for our experiments is available as https://github.com/tfjgeorge/ntk_alignment.
APA
Baratin, A., George, T., Laurent, C., Devon Hjelm, R., Lajoie, G., Vincent, P. & Lacoste-Julien, S.. (2021). Implicit Regularization via Neural Feature Alignment . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:2269-2277 Available from https://proceedings.mlr.press/v130/baratin21a.html.

Related Material