On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

Sanjeev Arora, Nadav Cohen, Elad Hazan
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:244-253, 2018.

Abstract

Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization. This paper suggests that, sometimes, increasing depth can speed up optimization. The effect of depth on optimization is decoupled from expressiveness by focusing on settings where additional layers amount to overparameterization – linear neural networks, a well-studied model. Theoretical analysis, as well as experiments, show that here depth acts as a preconditioner which may accelerate convergence. Even on simple convex problems such as linear regression with $\ell_p$ loss, $p>2$, gradient descent can benefit from transitioning to a non-convex overparameterized objective, more than it would from some common acceleration schemes. We also prove that it is mathematically impossible to obtain the acceleration effect of overparametrization via gradients of any regularizer.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-arora18a, title = {On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization}, author = {Arora, Sanjeev and Cohen, Nadav and Hazan, Elad}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {244--253}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/arora18a/arora18a.pdf}, url = {https://proceedings.mlr.press/v80/arora18a.html}, abstract = {Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization. This paper suggests that, sometimes, increasing depth can speed up optimization. The effect of depth on optimization is decoupled from expressiveness by focusing on settings where additional layers amount to overparameterization – linear neural networks, a well-studied model. Theoretical analysis, as well as experiments, show that here depth acts as a preconditioner which may accelerate convergence. Even on simple convex problems such as linear regression with $\ell_p$ loss, $p>2$, gradient descent can benefit from transitioning to a non-convex overparameterized objective, more than it would from some common acceleration schemes. We also prove that it is mathematically impossible to obtain the acceleration effect of overparametrization via gradients of any regularizer.} }
Endnote
%0 Conference Paper %T On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization %A Sanjeev Arora %A Nadav Cohen %A Elad Hazan %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-arora18a %I PMLR %P 244--253 %U https://proceedings.mlr.press/v80/arora18a.html %V 80 %X Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization. This paper suggests that, sometimes, increasing depth can speed up optimization. The effect of depth on optimization is decoupled from expressiveness by focusing on settings where additional layers amount to overparameterization – linear neural networks, a well-studied model. Theoretical analysis, as well as experiments, show that here depth acts as a preconditioner which may accelerate convergence. Even on simple convex problems such as linear regression with $\ell_p$ loss, $p>2$, gradient descent can benefit from transitioning to a non-convex overparameterized objective, more than it would from some common acceleration schemes. We also prove that it is mathematically impossible to obtain the acceleration effect of overparametrization via gradients of any regularizer.
APA
Arora, S., Cohen, N. & Hazan, E.. (2018). On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:244-253 Available from https://proceedings.mlr.press/v80/arora18a.html.

Related Material