Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach

Setareh Ariafar, Justin Gilmer, Zachary Nado, Jasper Snoek, Rodolphe Jenatton, George Dahl
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:11056-11071, 2022.

Abstract

Black box optimization requires specifying a search space to explore for solutions, e.g. a d-dimensional compact space, and this choice is critical for getting the best results at a reasonable budget. Unfortunately, determining a high quality search space can be challenging in many applications. For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable. The goal of this work is to motivate—through example applications in tuning deep neural networks—the problem of predicting the quality of search spaces conditioned on budgets, as well as to provide a simple scoring method based on a utility function applied to a probabilistic response surface model, similar to Bayesian optimization. We show that the method we present can compute meaningful budget-conditional scores in a variety of situations. We also provide experimental evidence that accurate scores can be useful in constructing and pruning search spaces. Ultimately, we believe scoring search spaces should become standard practice in the experimental workflow for deep learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-ariafar22a, title = { Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach }, author = {Ariafar, Setareh and Gilmer, Justin and Nado, Zachary and Snoek, Jasper and Jenatton, Rodolphe and Dahl, George}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {11056--11071}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/ariafar22a/ariafar22a.pdf}, url = {https://proceedings.mlr.press/v151/ariafar22a.html}, abstract = { Black box optimization requires specifying a search space to explore for solutions, e.g. a d-dimensional compact space, and this choice is critical for getting the best results at a reasonable budget. Unfortunately, determining a high quality search space can be challenging in many applications. For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable. The goal of this work is to motivate—through example applications in tuning deep neural networks—the problem of predicting the quality of search spaces conditioned on budgets, as well as to provide a simple scoring method based on a utility function applied to a probabilistic response surface model, similar to Bayesian optimization. We show that the method we present can compute meaningful budget-conditional scores in a variety of situations. We also provide experimental evidence that accurate scores can be useful in constructing and pruning search spaces. Ultimately, we believe scoring search spaces should become standard practice in the experimental workflow for deep learning. } }
Endnote
%0 Conference Paper %T Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach %A Setareh Ariafar %A Justin Gilmer %A Zachary Nado %A Jasper Snoek %A Rodolphe Jenatton %A George Dahl %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-ariafar22a %I PMLR %P 11056--11071 %U https://proceedings.mlr.press/v151/ariafar22a.html %V 151 %X Black box optimization requires specifying a search space to explore for solutions, e.g. a d-dimensional compact space, and this choice is critical for getting the best results at a reasonable budget. Unfortunately, determining a high quality search space can be challenging in many applications. For example, when tuning hyperparameters for machine learning pipelines on a new problem given a limited budget, one must strike a balance between excluding potentially promising regions and keeping the search space small enough to be tractable. The goal of this work is to motivate—through example applications in tuning deep neural networks—the problem of predicting the quality of search spaces conditioned on budgets, as well as to provide a simple scoring method based on a utility function applied to a probabilistic response surface model, similar to Bayesian optimization. We show that the method we present can compute meaningful budget-conditional scores in a variety of situations. We also provide experimental evidence that accurate scores can be useful in constructing and pruning search spaces. Ultimately, we believe scoring search spaces should become standard practice in the experimental workflow for deep learning.
APA
Ariafar, S., Gilmer, J., Nado, Z., Snoek, J., Jenatton, R. & Dahl, G.. (2022). Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:11056-11071 Available from https://proceedings.mlr.press/v151/ariafar22a.html.

Related Material