Compressing tree ensembles through Level-wise Optimization and Pruning

Laurens Devos, Timo Martens, Deniz Can Oruc, Wannes Meert, Hendrik Blockeel, Jesse Davis
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:13419-13443, 2025.

Abstract

Tree ensembles (e.g., gradient boosting decision trees) are often used in practice because they offer excellent predictive performance while still being easy and efficient to learn. In some contexts, it is important to additionally optimize their size: this is specifically the case when models need to have verifiable properties (verification of fairness, robustness, etc. is often exponential in the ensemble’s size), or when models run on battery-powered devices (smaller ensembles consume less energy, increasing battery autonomy). For this reason, compression of tree ensembles is worth studying. This paper presents LOP, a method for compressing a given tree ensemble by pruning or entirely removing trees in it, while updating leaf predictions in such a way that predictive accuracy is mostly unaffected. Empirically, LOP achieves compression factors that are often 10 to 100 times better than that of competing methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-devos25a, title = {Compressing tree ensembles through Level-wise Optimization and Pruning}, author = {Devos, Laurens and Martens, Timo and Oruc, Deniz Can and Meert, Wannes and Blockeel, Hendrik and Davis, Jesse}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {13419--13443}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/devos25a/devos25a.pdf}, url = {https://proceedings.mlr.press/v267/devos25a.html}, abstract = {Tree ensembles (e.g., gradient boosting decision trees) are often used in practice because they offer excellent predictive performance while still being easy and efficient to learn. In some contexts, it is important to additionally optimize their size: this is specifically the case when models need to have verifiable properties (verification of fairness, robustness, etc. is often exponential in the ensemble’s size), or when models run on battery-powered devices (smaller ensembles consume less energy, increasing battery autonomy). For this reason, compression of tree ensembles is worth studying. This paper presents LOP, a method for compressing a given tree ensemble by pruning or entirely removing trees in it, while updating leaf predictions in such a way that predictive accuracy is mostly unaffected. Empirically, LOP achieves compression factors that are often 10 to 100 times better than that of competing methods.} }
Endnote
%0 Conference Paper %T Compressing tree ensembles through Level-wise Optimization and Pruning %A Laurens Devos %A Timo Martens %A Deniz Can Oruc %A Wannes Meert %A Hendrik Blockeel %A Jesse Davis %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-devos25a %I PMLR %P 13419--13443 %U https://proceedings.mlr.press/v267/devos25a.html %V 267 %X Tree ensembles (e.g., gradient boosting decision trees) are often used in practice because they offer excellent predictive performance while still being easy and efficient to learn. In some contexts, it is important to additionally optimize their size: this is specifically the case when models need to have verifiable properties (verification of fairness, robustness, etc. is often exponential in the ensemble’s size), or when models run on battery-powered devices (smaller ensembles consume less energy, increasing battery autonomy). For this reason, compression of tree ensembles is worth studying. This paper presents LOP, a method for compressing a given tree ensemble by pruning or entirely removing trees in it, while updating leaf predictions in such a way that predictive accuracy is mostly unaffected. Empirically, LOP achieves compression factors that are often 10 to 100 times better than that of competing methods.
APA
Devos, L., Martens, T., Oruc, D.C., Meert, W., Blockeel, H. & Davis, J.. (2025). Compressing tree ensembles through Level-wise Optimization and Pruning. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:13419-13443 Available from https://proceedings.mlr.press/v267/devos25a.html.

Related Material