Searching in the Forest for Local Bayesian Optimization

Difan Deng, Marius Lindauer
ECMLPKDD Workshop on Meta-Knowledge Transfer, PMLR 191:38-50, 2022.

Abstract

Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach in dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments showed that the loss landscapes of HPO problems tend to be more benign than previously assumed, i.e. in the best case uni-modal and convex, such that a BO framework could be more efficient if it can focus on those promising local regions. In this paper, we propose BOinG, a two-stage approach that is tailored toward HPO problems. In the first stage, we build a scalable global surrogate model with a random forest to describe the overall landscape structure. Further, we choose a promising subregion via a bottom-up approach to the upper-level tree structure. In the second stage, a local model in this subregion is utilized to suggest the point to be evaluated next. Empirical experiments show that BOinG is able to exploit the structure of typical HPO problems and performs particularly well on various problems from synthetic functions and HPO.

Cite this Paper

BibTeX
@InProceedings{pmlr-v191-deng22a, title = {Searching in the Forest for Local Bayesian Optimization}, author = {Deng, Difan and Lindauer, Marius}, booktitle = {ECMLPKDD Workshop on Meta-Knowledge Transfer}, pages = {38--50}, year = {2022}, editor = {Brazdil, Pavel and van Rijn, Jan N. and Gouk, Henry and Mohr, Felix}, volume = {191}, series = {Proceedings of Machine Learning Research}, month = {23 Sep}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v191/deng22a/deng22a.pdf}, url = {https://proceedings.mlr.press/v191/deng22a.html}, abstract = {Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach in dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments showed that the loss landscapes of HPO problems tend to be more benign than previously assumed, i.e. in the best case uni-modal and convex, such that a BO framework could be more efficient if it can focus on those promising local regions. In this paper, we propose BOinG, a two-stage approach that is tailored toward HPO problems. In the first stage, we build a scalable global surrogate model with a random forest to describe the overall landscape structure. Further, we choose a promising subregion via a bottom-up approach to the upper-level tree structure. In the second stage, a local model in this subregion is utilized to suggest the point to be evaluated next. Empirical experiments show that BOinG is able to exploit the structure of typical HPO problems and performs particularly well on various problems from synthetic functions and HPO.} }
Endnote
%0 Conference Paper %T Searching in the Forest for Local Bayesian Optimization %A Difan Deng %A Marius Lindauer %B ECMLPKDD Workshop on Meta-Knowledge Transfer %C Proceedings of Machine Learning Research %D 2022 %E Pavel Brazdil %E Jan N. van Rijn %E Henry Gouk %E Felix Mohr %F pmlr-v191-deng22a %I PMLR %P 38--50 %U https://proceedings.mlr.press/v191/deng22a.html %V 191 %X Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach in dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments showed that the loss landscapes of HPO problems tend to be more benign than previously assumed, i.e. in the best case uni-modal and convex, such that a BO framework could be more efficient if it can focus on those promising local regions. In this paper, we propose BOinG, a two-stage approach that is tailored toward HPO problems. In the first stage, we build a scalable global surrogate model with a random forest to describe the overall landscape structure. Further, we choose a promising subregion via a bottom-up approach to the upper-level tree structure. In the second stage, a local model in this subregion is utilized to suggest the point to be evaluated next. Empirical experiments show that BOinG is able to exploit the structure of typical HPO problems and performs particularly well on various problems from synthetic functions and HPO.
APA
Deng, D. & Lindauer, M.. (2022). Searching in the Forest for Local Bayesian Optimization. ECMLPKDD Workshop on Meta-Knowledge Transfer, in Proceedings of Machine Learning Research 191:38-50 Available from https://proceedings.mlr.press/v191/deng22a.html.

Related Material