Characterizing Implicit Bias in Terms of Optimization Geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1832-1841, 2018.

Abstract

We study the bias of generic optimization methods, including Mirror Descent, Natural Gradient Descent and Steepest Descent with respect to different potentials and norms, when optimizing underdetermined linear models or separable linear classification problems. We ask the question of whether the global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm, and indecently of hyper-parameter choices, such as stepsize and momentum.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-gunasekar18a, title = {Characterizing Implicit Bias in Terms of Optimization Geometry}, author = {Gunasekar, Suriya and Lee, Jason and Soudry, Daniel and Srebro, Nathan}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {1832--1841}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/gunasekar18a/gunasekar18a.pdf}, url = {https://proceedings.mlr.press/v80/gunasekar18a.html}, abstract = {We study the bias of generic optimization methods, including Mirror Descent, Natural Gradient Descent and Steepest Descent with respect to different potentials and norms, when optimizing underdetermined linear models or separable linear classification problems. We ask the question of whether the global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm, and indecently of hyper-parameter choices, such as stepsize and momentum.} }
Endnote
%0 Conference Paper %T Characterizing Implicit Bias in Terms of Optimization Geometry %A Suriya Gunasekar %A Jason Lee %A Daniel Soudry %A Nathan Srebro %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-gunasekar18a %I PMLR %P 1832--1841 %U https://proceedings.mlr.press/v80/gunasekar18a.html %V 80 %X We study the bias of generic optimization methods, including Mirror Descent, Natural Gradient Descent and Steepest Descent with respect to different potentials and norms, when optimizing underdetermined linear models or separable linear classification problems. We ask the question of whether the global minimum (among the many possible global minima) reached by optimization can be characterized in terms of the potential or norm, and indecently of hyper-parameter choices, such as stepsize and momentum.
APA
Gunasekar, S., Lee, J., Soudry, D. & Srebro, N.. (2018). Characterizing Implicit Bias in Terms of Optimization Geometry. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:1832-1841 Available from https://proceedings.mlr.press/v80/gunasekar18a.html.

Related Material