We Still Don’t Understand High-Dimensional Bayesian Optimization

Colin Doumont, Donney Fan, Natalie Maus, Jacob R. Gardner, Henry Moss, Geoff Pleiss
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2629-2637, 2026.

Abstract

Existing high-dimensional Bayesian optimization (BO) methods aim to overcome the curse of dimensionality by carefully encoding structural assumptions, from locality to sparsity to smoothness, into the optimization procedure. Surprisingly, we demonstrate that these approaches are outperformed by arguably the simplest method imaginable: Bayesian linear regression. After applying a geometric transformation to avoid boundary-seeking behavior, Gaussian processes with linear kernels match state-of-the-art performance on tasks with 60- to 6,000-dimensional search spaces. Linear models offer numerous advantages over their non-parametric counterparts: they afford closed-form sampling and their computation scales linearly with data, a fact we exploit on molecular optimization tasks with >20,000 observations. Coupled with empirical analyses, our results suggest the need to depart from past intuitions about BO methods in high-dimensions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-doumont26a, title = { We Still Don’t Understand High-Dimensional Bayesian Optimization }, author = {Doumont, Colin and Fan, Donney and Maus, Natalie and Gardner, Jacob R. and Moss, Henry and Pleiss, Geoff}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2629--2637}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/doumont26a/doumont26a.pdf}, url = {https://proceedings.mlr.press/v300/doumont26a.html}, abstract = { Existing high-dimensional Bayesian optimization (BO) methods aim to overcome the curse of dimensionality by carefully encoding structural assumptions, from locality to sparsity to smoothness, into the optimization procedure. Surprisingly, we demonstrate that these approaches are outperformed by arguably the simplest method imaginable: Bayesian linear regression. After applying a geometric transformation to avoid boundary-seeking behavior, Gaussian processes with linear kernels match state-of-the-art performance on tasks with 60- to 6,000-dimensional search spaces. Linear models offer numerous advantages over their non-parametric counterparts: they afford closed-form sampling and their computation scales linearly with data, a fact we exploit on molecular optimization tasks with >20,000 observations. Coupled with empirical analyses, our results suggest the need to depart from past intuitions about BO methods in high-dimensions. } }
Endnote
%0 Conference Paper %T We Still Don’t Understand High-Dimensional Bayesian Optimization %A Colin Doumont %A Donney Fan %A Natalie Maus %A Jacob R. Gardner %A Henry Moss %A Geoff Pleiss %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-doumont26a %I PMLR %P 2629--2637 %U https://proceedings.mlr.press/v300/doumont26a.html %V 300 %X Existing high-dimensional Bayesian optimization (BO) methods aim to overcome the curse of dimensionality by carefully encoding structural assumptions, from locality to sparsity to smoothness, into the optimization procedure. Surprisingly, we demonstrate that these approaches are outperformed by arguably the simplest method imaginable: Bayesian linear regression. After applying a geometric transformation to avoid boundary-seeking behavior, Gaussian processes with linear kernels match state-of-the-art performance on tasks with 60- to 6,000-dimensional search spaces. Linear models offer numerous advantages over their non-parametric counterparts: they afford closed-form sampling and their computation scales linearly with data, a fact we exploit on molecular optimization tasks with >20,000 observations. Coupled with empirical analyses, our results suggest the need to depart from past intuitions about BO methods in high-dimensions.
APA
Doumont, C., Fan, D., Maus, N., Gardner, J.R., Moss, H. & Pleiss, G.. (2026). We Still Don’t Understand High-Dimensional Bayesian Optimization . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2629-2637 Available from https://proceedings.mlr.press/v300/doumont26a.html.

Related Material