Sequential Model-Based Ensemble Optimization

Alexandre Lacoste Laval University, Hugo Larochelle, Mario Marchand Laval University, François Laviolette Laval University
Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, PMLR R12:706-714, 2014.

Abstract

One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as Agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR12-university14t, title = {Sequential Model-Based Ensemble Optimization}, author = {University, Alexandre Lacoste Laval and Larochelle, Hugo and University, Mario Marchand Laval and University, Fran{\c{c}}ois Laviolette Laval}, booktitle = {Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence}, pages = {706--714}, year = {2014}, editor = {Zhang, Nevin L. and Tian, Jin}, volume = {R12}, series = {Proceedings of Machine Learning Research}, month = {23--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r12/main/assets/university14t/university14t.pdf}, url = {https://proceedings.mlr.press/r12/university14t.html}, abstract = {One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as Agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Sequential Model-Based Ensemble Optimization %A Alexandre Lacoste Laval University %A Hugo Larochelle %A Mario Marchand Laval University %A François Laviolette Laval University %B Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2014 %E Nevin L. Zhang %E Jin Tian %F pmlr-vR12-university14t %I PMLR %P 706--714 %U https://proceedings.mlr.press/r12/university14t.html %V R12 %X One of the most tedious tasks in the application of machine learning is model selection, i.e. hyperparameter selection. Fortunately, recent progress has been made in the automation of this process, through the use of sequential model-based optimization (SMBO) methods. This can be used to optimize a cross-validation performance of a learning algorithm over the value of its hyperparameters. However, it is well known that ensembles of learned models almost consistently outperform a single model, even if properly selected. In this paper, we thus propose an extension of SMBO methods that automatically constructs such ensembles. This method builds on a recently proposed ensemble construction paradigm known as Agnostic Bayesian learning. In experiments on 22 regression and 39 classification data sets, we confirm the success of this proposed approach, which is able to outperform model selection with SMBO. %Z Reissued by PMLR on 04 October 2026.
APA
University, A.L.L., Larochelle, H., University, M.M.L. & University, F.L.L.. (2014). Sequential Model-Based Ensemble Optimization. Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R12:706-714 Available from https://proceedings.mlr.press/r12/university14t.html. Reissued by PMLR on 04 October 2026.

Related Material