Learning to Rank Learning Curves

Martin Wistuba, Tejaswini Pedapati
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10303-10312, 2020.

Abstract

Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-wistuba20a, title = {Learning to Rank Learning Curves}, author = {Wistuba, Martin and Pedapati, Tejaswini}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10303--10312}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/wistuba20a/wistuba20a.pdf}, url = {https://proceedings.mlr.press/v119/wistuba20a.html}, abstract = {Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.} }
Endnote
%0 Conference Paper %T Learning to Rank Learning Curves %A Martin Wistuba %A Tejaswini Pedapati %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-wistuba20a %I PMLR %P 10303--10312 %U https://proceedings.mlr.press/v119/wistuba20a.html %V 119 %X Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.
APA
Wistuba, M. & Pedapati, T.. (2020). Learning to Rank Learning Curves. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10303-10312 Available from https://proceedings.mlr.press/v119/wistuba20a.html.

Related Material