Bayesian Optimisation over Multiple Continuous and Categorical Inputs

Binxin Ru, Ahsan Alvi, Vu Nguyen, Michael A. Osborne, Stephen Roberts
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:8276-8285, 2020.

Abstract

Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. Current approaches, like one-hot encoding, severely increase the dimension of the search space, while separate modelling of category-specific data is sample-inefficient. Both frameworks are not scalable to practical applications involving multiple categorical variables, each with multiple possible values. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-ru20a, title = {{B}ayesian Optimisation over Multiple Continuous and Categorical Inputs}, author = {Ru, Binxin and Alvi, Ahsan and Nguyen, Vu and Osborne, Michael A. and Roberts, Stephen}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {8276--8285}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/ru20a/ru20a.pdf}, url = {https://proceedings.mlr.press/v119/ru20a.html}, abstract = {Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. Current approaches, like one-hot encoding, severely increase the dimension of the search space, while separate modelling of category-specific data is sample-inefficient. Both frameworks are not scalable to practical applications involving multiple categorical variables, each with multiple possible values. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.} }
Endnote
%0 Conference Paper %T Bayesian Optimisation over Multiple Continuous and Categorical Inputs %A Binxin Ru %A Ahsan Alvi %A Vu Nguyen %A Michael A. Osborne %A Stephen Roberts %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-ru20a %I PMLR %P 8276--8285 %U https://proceedings.mlr.press/v119/ru20a.html %V 119 %X Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. Current approaches, like one-hot encoding, severely increase the dimension of the search space, while separate modelling of category-specific data is sample-inefficient. Both frameworks are not scalable to practical applications involving multiple categorical variables, each with multiple possible values. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.
APA
Ru, B., Alvi, A., Nguyen, V., Osborne, M.A. & Roberts, S.. (2020). Bayesian Optimisation over Multiple Continuous and Categorical Inputs. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:8276-8285 Available from https://proceedings.mlr.press/v119/ru20a.html.

Related Material