Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach

Syrine Belakaria, Janardhan Rao Doppa, Nicolo Fusi, Rishit Sheth
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:9076-9093, 2023.

Abstract

The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as Budget-Aware Planning for Iterative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-belakaria23a, title = {Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach}, author = {Belakaria, Syrine and Doppa, Janardhan Rao and Fusi, Nicolo and Sheth, Rishit}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {9076--9093}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/belakaria23a/belakaria23a.pdf}, url = {https://proceedings.mlr.press/v206/belakaria23a.html}, abstract = {The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as Budget-Aware Planning for Iterative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.} }
Endnote
%0 Conference Paper %T Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach %A Syrine Belakaria %A Janardhan Rao Doppa %A Nicolo Fusi %A Rishit Sheth %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-belakaria23a %I PMLR %P 9076--9093 %U https://proceedings.mlr.press/v206/belakaria23a.html %V 206 %X The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as Budget-Aware Planning for Iterative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.
APA
Belakaria, S., Doppa, J.R., Fusi, N. & Sheth, R.. (2023). Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:9076-9093 Available from https://proceedings.mlr.press/v206/belakaria23a.html.

Related Material