Lexicographic and DepthSensitive Margins in Homogeneous and NonHomogeneous Deep Models
[edit]
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:46834692, 2019.
Abstract
With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the “constrained path”), relate it to the limit of a “margin path” and characterize the resulting solution. For nonhomogeneous ensemble models, which output is a sum of homogeneous submodels, we show that this solution discards the shallowest submodels if they are unnecessary. For homogeneous models, we show convergence to a “lexicographic maxmargin solution”, and provide conditions under which maxmargin solutions are also attained as the limit of unconstrained gradient descent.
Related Material


