Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, Daniel Soudry
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4683-4692, 2019.

Abstract

With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the “constrained path”), relate it to the limit of a “margin path” and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a “lexicographic max-margin solution”, and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-nacson19a, title = {Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models}, author = {Nacson, Mor Shpigel and Gunasekar, Suriya and Lee, Jason and Srebro, Nathan and Soudry, Daniel}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {4683--4692}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/nacson19a/nacson19a.pdf}, url = {https://proceedings.mlr.press/v97/nacson19a.html}, abstract = {With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the “constrained path”), relate it to the limit of a “margin path” and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a “lexicographic max-margin solution”, and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.} }
Endnote
%0 Conference Paper %T Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models %A Mor Shpigel Nacson %A Suriya Gunasekar %A Jason Lee %A Nathan Srebro %A Daniel Soudry %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-nacson19a %I PMLR %P 4683--4692 %U https://proceedings.mlr.press/v97/nacson19a.html %V 97 %X With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the “constrained path”), relate it to the limit of a “margin path” and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a “lexicographic max-margin solution”, and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.
APA
Nacson, M.S., Gunasekar, S., Lee, J., Srebro, N. & Soudry, D.. (2019). Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:4683-4692 Available from https://proceedings.mlr.press/v97/nacson19a.html.

Related Material