Learn2Hop: Learned Optimization on Rough Landscapes

Amil Merchant, Luke Metz, Samuel S Schoenholz, Ekin D Cubuk
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:7643-7653, 2021.

Abstract

Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material design. Yet, current techniques either require extremely high iteration counts or a large number of random restarts for good performance. In this work, we propose adapting recent developments in meta-learning to these many-minima problems by learning the optimization algorithm for various loss landscapes. We focus on problems from atomic structural optimization—finding low energy configurations of many-atom systems—including widely studied models such as bimetallic clusters and disordered silicon. We find that our optimizer learns a hopping behavior which enables efficient exploration and improves the rate of low energy minima discovery. Finally, our learned optimizers show promising generalization with efficiency gains on never before seen tasks (e.g. new elements or compositions). Code is available at https://learn2hop.page.link/github.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-merchant21a, title = {Learn2Hop: Learned Optimization on Rough Landscapes}, author = {Merchant, Amil and Metz, Luke and Schoenholz, Samuel S and Cubuk, Ekin D}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {7643--7653}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/merchant21a/merchant21a.pdf}, url = {https://proceedings.mlr.press/v139/merchant21a.html}, abstract = {Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material design. Yet, current techniques either require extremely high iteration counts or a large number of random restarts for good performance. In this work, we propose adapting recent developments in meta-learning to these many-minima problems by learning the optimization algorithm for various loss landscapes. We focus on problems from atomic structural optimization—finding low energy configurations of many-atom systems—including widely studied models such as bimetallic clusters and disordered silicon. We find that our optimizer learns a hopping behavior which enables efficient exploration and improves the rate of low energy minima discovery. Finally, our learned optimizers show promising generalization with efficiency gains on never before seen tasks (e.g. new elements or compositions). Code is available at https://learn2hop.page.link/github.} }
Endnote
%0 Conference Paper %T Learn2Hop: Learned Optimization on Rough Landscapes %A Amil Merchant %A Luke Metz %A Samuel S Schoenholz %A Ekin D Cubuk %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-merchant21a %I PMLR %P 7643--7653 %U https://proceedings.mlr.press/v139/merchant21a.html %V 139 %X Optimization of non-convex loss surfaces containing many local minima remains a critical problem in a variety of domains, including operations research, informatics, and material design. Yet, current techniques either require extremely high iteration counts or a large number of random restarts for good performance. In this work, we propose adapting recent developments in meta-learning to these many-minima problems by learning the optimization algorithm for various loss landscapes. We focus on problems from atomic structural optimization—finding low energy configurations of many-atom systems—including widely studied models such as bimetallic clusters and disordered silicon. We find that our optimizer learns a hopping behavior which enables efficient exploration and improves the rate of low energy minima discovery. Finally, our learned optimizers show promising generalization with efficiency gains on never before seen tasks (e.g. new elements or compositions). Code is available at https://learn2hop.page.link/github.
APA
Merchant, A., Metz, L., Schoenholz, S.S. & Cubuk, E.D.. (2021). Learn2Hop: Learned Optimization on Rough Landscapes. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:7643-7653 Available from https://proceedings.mlr.press/v139/merchant21a.html.

Related Material