High Dimensional Bayesian Optimisation and Bandits via Additive Models

Kirthevasan Kandasamy, Jeff Schneider, Barnabas Poczos
Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:295-304, 2015.

Abstract

Bayesian Optimisation (BO) is a technique used in optimising a D-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on D even though the function depends on all D dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.

Cite this Paper

BibTeX
@InProceedings{pmlr-v37-kandasamy15, title = {High Dimensional Bayesian Optimisation and Bandits via Additive Models}, author = {Kandasamy, Kirthevasan and Schneider, Jeff and Poczos, Barnabas}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning}, pages = {295--304}, year = {2015}, editor = {Bach, Francis and Blei, David}, volume = {37}, series = {Proceedings of Machine Learning Research}, address = {Lille, France}, month = {07--09 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v37/kandasamy15.pdf}, url = {https://proceedings.mlr.press/v37/kandasamy15.html}, abstract = {Bayesian Optimisation (BO) is a technique used in optimising a D-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on D even though the function depends on all D dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.} }
Endnote
%0 Conference Paper %T High Dimensional Bayesian Optimisation and Bandits via Additive Models %A Kirthevasan Kandasamy %A Jeff Schneider %A Barnabas Poczos %B Proceedings of the 32nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2015 %E Francis Bach %E David Blei %F pmlr-v37-kandasamy15 %I PMLR %P 295--304 %U https://proceedings.mlr.press/v37/kandasamy15.html %V 37 %X Bayesian Optimisation (BO) is a technique used in optimising a D-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on D even though the function depends on all D dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.
RIS
TY - CPAPER TI - High Dimensional Bayesian Optimisation and Bandits via Additive Models AU - Kirthevasan Kandasamy AU - Jeff Schneider AU - Barnabas Poczos BT - Proceedings of the 32nd International Conference on Machine Learning DA - 2015/06/01 ED - Francis Bach ED - David Blei ID - pmlr-v37-kandasamy15 PB - PMLR DP - Proceedings of Machine Learning Research VL - 37 SP - 295 EP - 304 L1 - http://proceedings.mlr.press/v37/kandasamy15.pdf UR - https://proceedings.mlr.press/v37/kandasamy15.html AB - Bayesian Optimisation (BO) is a technique used in optimising a D-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on D even though the function depends on all D dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive. ER -
APA
Kandasamy, K., Schneider, J. & Poczos, B.. (2015). High Dimensional Bayesian Optimisation and Bandits via Additive Models. Proceedings of the 32nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 37:295-304 Available from https://proceedings.mlr.press/v37/kandasamy15.html.

Related Material