Non-Linear Gradient Boosting for Class-Imbalance Learning

Jordan Frery, Amaury Habrard, Marc Sebban, Liyun He-Guelton
Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, PMLR 94:38-51, 2018.

Abstract

Gradient boosting relies on linearly combining diverse and weak hypotheses to build a strong classifier. In the class imbalance setting, boosting algorithms often require many hypotheses which tend to be more complex and may increase the risk of overfitting. We propose in this paper to address this issue by adapting the gradient boosting framework to a non-linear setting. In order to learn the idiosyncrasies of the target concept and prevent the algorithm from being biased toward the majority class, we suggest to jointly learn different combinations of the same set of very weak classifiers and expand the expressiveness of the final model by leveraging their non-linear complementarity. We perform an extensive experimental study using decision trees and show that, while requiring much less weak learners with a lower complexity (fewer splits per tree), our model outperforms standard linear gradient boosting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v94-frery18a, title = {Non-Linear Gradient Boosting for Class-Imbalance Learning}, author = {Frery, Jordan and Habrard, Amaury and Sebban, Marc and He-Guelton, Liyun}, booktitle = {Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications}, pages = {38--51}, year = {2018}, editor = {Torgo, Luís and Matwin, Stan and Japkowicz, Nathalie and Krawczyk, Bartosz and Moniz, Nuno and Branco, Paula}, volume = {94}, series = {Proceedings of Machine Learning Research}, month = {10 Sep}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v94/frery18a/frery18a.pdf}, url = {https://proceedings.mlr.press/v94/frery18a.html}, abstract = {Gradient boosting relies on linearly combining diverse and weak hypotheses to build a strong classifier. In the class imbalance setting, boosting algorithms often require many hypotheses which tend to be more complex and may increase the risk of overfitting. We propose in this paper to address this issue by adapting the gradient boosting framework to a non-linear setting. In order to learn the idiosyncrasies of the target concept and prevent the algorithm from being biased toward the majority class, we suggest to jointly learn different combinations of the same set of very weak classifiers and expand the expressiveness of the final model by leveraging their non-linear complementarity. We perform an extensive experimental study using decision trees and show that, while requiring much less weak learners with a lower complexity (fewer splits per tree), our model outperforms standard linear gradient boosting.} }
Endnote
%0 Conference Paper %T Non-Linear Gradient Boosting for Class-Imbalance Learning %A Jordan Frery %A Amaury Habrard %A Marc Sebban %A Liyun He-Guelton %B Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications %C Proceedings of Machine Learning Research %D 2018 %E Luís Torgo %E Stan Matwin %E Nathalie Japkowicz %E Bartosz Krawczyk %E Nuno Moniz %E Paula Branco %F pmlr-v94-frery18a %I PMLR %P 38--51 %U https://proceedings.mlr.press/v94/frery18a.html %V 94 %X Gradient boosting relies on linearly combining diverse and weak hypotheses to build a strong classifier. In the class imbalance setting, boosting algorithms often require many hypotheses which tend to be more complex and may increase the risk of overfitting. We propose in this paper to address this issue by adapting the gradient boosting framework to a non-linear setting. In order to learn the idiosyncrasies of the target concept and prevent the algorithm from being biased toward the majority class, we suggest to jointly learn different combinations of the same set of very weak classifiers and expand the expressiveness of the final model by leveraging their non-linear complementarity. We perform an extensive experimental study using decision trees and show that, while requiring much less weak learners with a lower complexity (fewer splits per tree), our model outperforms standard linear gradient boosting.
APA
Frery, J., Habrard, A., Sebban, M. & He-Guelton, L.. (2018). Non-Linear Gradient Boosting for Class-Imbalance Learning. Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, in Proceedings of Machine Learning Research 94:38-51 Available from https://proceedings.mlr.press/v94/frery18a.html.

Related Material