Bayesian Hyperparameter Optimization for Ensemble Learning

Julien-Charles Levesque, Christian Gagne, Robert Sabourin Ecole de Technologie Superieure
Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, PMLR R14:198-207, 2016.

Abstract

In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-size ensemble, optimizing the configuration of one classifier of the ensemble at each iteration of the hyperparameter optimization algorithm, taking into consideration the interaction with the other models when evaluating potential performances. We also consider the case where the ensemble is to be reconstructed at the end of the hyperparameter optimization phase, through a greedy selection over the pool of models generated during the optimization. We study the performance of our proposed method on three different hyperparameter spaces, showing that our approach is better than both the best single model and a greedy ensemble construction over the models produced by a standard Bayesian optimization.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR14-levesque16a, title = {{B}ayesian Hyperparameter Optimization for Ensemble Learning}, author = {Levesque, Julien-Charles and Gagne, Christian and Superieure, Robert Sabourin Ecole de Technologie}, booktitle = {Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence}, pages = {198--207}, year = {2016}, editor = {Ihler, Alexander and Janzing, Dominik}, volume = {R14}, series = {Proceedings of Machine Learning Research}, month = {25--29 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r14/main/assets/levesque16a/levesque16a.pdf}, url = {https://proceedings.mlr.press/r14/levesque16a.html}, abstract = {In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-size ensemble, optimizing the configuration of one classifier of the ensemble at each iteration of the hyperparameter optimization algorithm, taking into consideration the interaction with the other models when evaluating potential performances. We also consider the case where the ensemble is to be reconstructed at the end of the hyperparameter optimization phase, through a greedy selection over the pool of models generated during the optimization. We study the performance of our proposed method on three different hyperparameter spaces, showing that our approach is better than both the best single model and a greedy ensemble construction over the models produced by a standard Bayesian optimization.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Bayesian Hyperparameter Optimization for Ensemble Learning %A Julien-Charles Levesque %A Christian Gagne %A Robert Sabourin Ecole de Technologie Superieure %B Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2016 %E Alexander Ihler %E Dominik Janzing %F pmlr-vR14-levesque16a %I PMLR %P 198--207 %U https://proceedings.mlr.press/r14/levesque16a.html %V R14 %X In this paper, we bridge the gap between hyperparameter optimization and ensemble learning by performing Bayesian optimization of an ensemble with regards to its hyperparameters. Our method consists in building a fixed-size ensemble, optimizing the configuration of one classifier of the ensemble at each iteration of the hyperparameter optimization algorithm, taking into consideration the interaction with the other models when evaluating potential performances. We also consider the case where the ensemble is to be reconstructed at the end of the hyperparameter optimization phase, through a greedy selection over the pool of models generated during the optimization. We study the performance of our proposed method on three different hyperparameter spaces, showing that our approach is better than both the best single model and a greedy ensemble construction over the models produced by a standard Bayesian optimization. %Z Reissued by PMLR on 04 October 2026.
APA
Levesque, J., Gagne, C. & Superieure, R.S.E.d.T.. (2016). Bayesian Hyperparameter Optimization for Ensemble Learning. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R14:198-207 Available from https://proceedings.mlr.press/r14/levesque16a.html. Reissued by PMLR on 04 October 2026.

Related Material