AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale

Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Ammanaghatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sukthanker, Frank Hutter, Euxhen Hasanaj, Tien-Dung Le, Mikhail Khodak, Yuriy Nevmyvaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, Evan Sparks
Proceedings of the NeurIPS 2022 Competitions Track, PMLR 220:151-170, 2022.

Abstract

The vision of Automated Machine Learning (AutoML) is to produce high performing ML pipelines that require very little human involvement or domain expertise to use. Competitions and benchmarks have been critical tools for accelerating progress in AutoML. However, much of the prior work on AutoML competitions has focused on well-studied domains in machine learning such as vision and language—these are domains which have benefited from several years of ML pipeline design by domain experts, which brings the usage of AutoML into question in the first place. Recently, AutoML for diverse tasks has emerged as an important research area that aims to bring AutoML to the domains where it can have the most impact: the long tail of ML tasks beyond vision and language. We present a retrospective report of the AutoML Decathlon—an AutoML for diverse tasks competition hosted at NeurIPS 2022. The AutoML Decathlon presented participants with a set of 10 machine learning tasks that are diverse along several axes: domain, input dimension, output dimension, output type, objective function, and scale. Participants were tasked with developing AutoML methods that performed well on a separate set of 10 hidden diverse test tasks within a certain time budget, so as to discourage overfitting to the initial set of tasks and to encourage efficiency. In this report, we outline the details of the competition, discuss the top-5 submissions, analyze the results, and compare top submissions to additional state-of-the-art baselines designed specifically for diverse tasks. We conclude that the combination of existing efficient AutoML techniques with modern advancements in ML such as large-scale transfer learning, modern architectures, and differentiable Neural Architecture Search (NAS) is a promising direction for AutoML for diverse tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v220-roberts23a, title = {AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale}, author = {Roberts, Nicholas and Guo, Samuel and Xu, Cong and Talwalkar, Ameet and Lander, David and Tao, Lvfang and Cai, Linhang and Niu, Shuaicheng and Heng, Jianyu and Qin, Hongyang and Deng, Minwen and Hog, Johannes and Pfefferle, Alexander and Shivakumar, Sushil Ammanaghatta and Krishnakumar, Arjun and Wang, Yubo and Sukthanker, Rhea and Hutter, Frank and Hasanaj, Euxhen and Le, Tien-Dung and Khodak, Mikhail and Nevmyvaka, Yuriy and Rasul, Kashif and Sala, Frederic and Schneider, Anderson and Shen, Junhong and Sparks, Evan}, booktitle = {Proceedings of the NeurIPS 2022 Competitions Track}, pages = {151--170}, year = {2022}, editor = {Ciccone, Marco and Stolovitzky, Gustavo and Albrecht, Jacob}, volume = {220}, series = {Proceedings of Machine Learning Research}, month = {28 Nov--09 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v220/roberts23a/roberts23a.pdf}, url = {https://proceedings.mlr.press/v220/roberts23a.html}, abstract = {The vision of Automated Machine Learning (AutoML) is to produce high performing ML pipelines that require very little human involvement or domain expertise to use. Competitions and benchmarks have been critical tools for accelerating progress in AutoML. However, much of the prior work on AutoML competitions has focused on well-studied domains in machine learning such as vision and language—these are domains which have benefited from several years of ML pipeline design by domain experts, which brings the usage of AutoML into question in the first place. Recently, AutoML for diverse tasks has emerged as an important research area that aims to bring AutoML to the domains where it can have the most impact: the long tail of ML tasks beyond vision and language. We present a retrospective report of the AutoML Decathlon—an AutoML for diverse tasks competition hosted at NeurIPS 2022. The AutoML Decathlon presented participants with a set of 10 machine learning tasks that are diverse along several axes: domain, input dimension, output dimension, output type, objective function, and scale. Participants were tasked with developing AutoML methods that performed well on a separate set of 10 hidden diverse test tasks within a certain time budget, so as to discourage overfitting to the initial set of tasks and to encourage efficiency. In this report, we outline the details of the competition, discuss the top-5 submissions, analyze the results, and compare top submissions to additional state-of-the-art baselines designed specifically for diverse tasks. We conclude that the combination of existing efficient AutoML techniques with modern advancements in ML such as large-scale transfer learning, modern architectures, and differentiable Neural Architecture Search (NAS) is a promising direction for AutoML for diverse tasks.} }
Endnote
%0 Conference Paper %T AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale %A Nicholas Roberts %A Samuel Guo %A Cong Xu %A Ameet Talwalkar %A David Lander %A Lvfang Tao %A Linhang Cai %A Shuaicheng Niu %A Jianyu Heng %A Hongyang Qin %A Minwen Deng %A Johannes Hog %A Alexander Pfefferle %A Sushil Ammanaghatta Shivakumar %A Arjun Krishnakumar %A Yubo Wang %A Rhea Sukthanker %A Frank Hutter %A Euxhen Hasanaj %A Tien-Dung Le %A Mikhail Khodak %A Yuriy Nevmyvaka %A Kashif Rasul %A Frederic Sala %A Anderson Schneider %A Junhong Shen %A Evan Sparks %B Proceedings of the NeurIPS 2022 Competitions Track %C Proceedings of Machine Learning Research %D 2022 %E Marco Ciccone %E Gustavo Stolovitzky %E Jacob Albrecht %F pmlr-v220-roberts23a %I PMLR %P 151--170 %U https://proceedings.mlr.press/v220/roberts23a.html %V 220 %X The vision of Automated Machine Learning (AutoML) is to produce high performing ML pipelines that require very little human involvement or domain expertise to use. Competitions and benchmarks have been critical tools for accelerating progress in AutoML. However, much of the prior work on AutoML competitions has focused on well-studied domains in machine learning such as vision and language—these are domains which have benefited from several years of ML pipeline design by domain experts, which brings the usage of AutoML into question in the first place. Recently, AutoML for diverse tasks has emerged as an important research area that aims to bring AutoML to the domains where it can have the most impact: the long tail of ML tasks beyond vision and language. We present a retrospective report of the AutoML Decathlon—an AutoML for diverse tasks competition hosted at NeurIPS 2022. The AutoML Decathlon presented participants with a set of 10 machine learning tasks that are diverse along several axes: domain, input dimension, output dimension, output type, objective function, and scale. Participants were tasked with developing AutoML methods that performed well on a separate set of 10 hidden diverse test tasks within a certain time budget, so as to discourage overfitting to the initial set of tasks and to encourage efficiency. In this report, we outline the details of the competition, discuss the top-5 submissions, analyze the results, and compare top submissions to additional state-of-the-art baselines designed specifically for diverse tasks. We conclude that the combination of existing efficient AutoML techniques with modern advancements in ML such as large-scale transfer learning, modern architectures, and differentiable Neural Architecture Search (NAS) is a promising direction for AutoML for diverse tasks.
APA
Roberts, N., Guo, S., Xu, C., Talwalkar, A., Lander, D., Tao, L., Cai, L., Niu, S., Heng, J., Qin, H., Deng, M., Hog, J., Pfefferle, A., Shivakumar, S.A., Krishnakumar, A., Wang, Y., Sukthanker, R., Hutter, F., Hasanaj, E., Le, T., Khodak, M., Nevmyvaka, Y., Rasul, K., Sala, F., Schneider, A., Shen, J. & Sparks, E.. (2022). AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale. Proceedings of the NeurIPS 2022 Competitions Track, in Proceedings of Machine Learning Research 220:151-170 Available from https://proceedings.mlr.press/v220/roberts23a.html.

Related Material