Low-loss connection of weight vectors: distribution-based approaches

Ivan Anokhin, Dmitry Yarotsky
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:335-344, 2020.

Abstract

Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel of methods used to connect two low-loss points by a low-loss curve on this surface. Our methods vary in accuracy and complexity. Most of our methods are based on ”macroscopic” distributional assumptions and are insensitive to the detailed properties of the points to be connected. Some methods require a prior training of a ”global connection model” which can then be applied to any pair of points. The accuracy of the method generally correlates with its complexity and sensitivity to the endpoint detail.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-anokhin20a, title = {Low-loss connection of weight vectors: distribution-based approaches}, author = {Anokhin, Ivan and Yarotsky, Dmitry}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {335--344}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/anokhin20a/anokhin20a.pdf}, url = {https://proceedings.mlr.press/v119/anokhin20a.html}, abstract = {Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel of methods used to connect two low-loss points by a low-loss curve on this surface. Our methods vary in accuracy and complexity. Most of our methods are based on ”macroscopic” distributional assumptions and are insensitive to the detailed properties of the points to be connected. Some methods require a prior training of a ”global connection model” which can then be applied to any pair of points. The accuracy of the method generally correlates with its complexity and sensitivity to the endpoint detail.} }
Endnote
%0 Conference Paper %T Low-loss connection of weight vectors: distribution-based approaches %A Ivan Anokhin %A Dmitry Yarotsky %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-anokhin20a %I PMLR %P 335--344 %U https://proceedings.mlr.press/v119/anokhin20a.html %V 119 %X Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel of methods used to connect two low-loss points by a low-loss curve on this surface. Our methods vary in accuracy and complexity. Most of our methods are based on ”macroscopic” distributional assumptions and are insensitive to the detailed properties of the points to be connected. Some methods require a prior training of a ”global connection model” which can then be applied to any pair of points. The accuracy of the method generally correlates with its complexity and sensitivity to the endpoint detail.
APA
Anokhin, I. & Yarotsky, D.. (2020). Low-loss connection of weight vectors: distribution-based approaches. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:335-344 Available from https://proceedings.mlr.press/v119/anokhin20a.html.

Related Material