GBHT: Gradient Boosting Histogram Transform for Density Estimation

Jingyi Cui, Hanyuan Hang, Yisen Wang, Zhouchen Lin
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2233-2243, 2021.

Abstract

In this paper, we propose a density estimation algorithm called \textit{Gradient Boosting Histogram Transform} (GBHT), where we adopt the \textit{Negative Log Likelihood} as the loss function to make the boosting procedure available for the unsupervised tasks. From a learning theory viewpoint, we first prove fast convergence rates for GBHT with the smoothness assumption that the underlying density function lies in the space $C^{0,\alpha}$. Then when the target density function lies in spaces $C^{1,\alpha}$, we present an upper bound for GBHT which is smaller than the lower bound of its corresponding base learner, in the sense of convergence rates. To the best of our knowledge, we make the first attempt to theoretically explain why boosting can enhance the performance of its base learners for density estimation problems. In experiments, we not only conduct performance comparisons with the widely used KDE, but also apply GBHT to anomaly detection to showcase a further application of GBHT.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-cui21c, title = {GBHT: Gradient Boosting Histogram Transform for Density Estimation}, author = {Cui, Jingyi and Hang, Hanyuan and Wang, Yisen and Lin, Zhouchen}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2233--2243}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/cui21c/cui21c.pdf}, url = {https://proceedings.mlr.press/v139/cui21c.html}, abstract = {In this paper, we propose a density estimation algorithm called \textit{Gradient Boosting Histogram Transform} (GBHT), where we adopt the \textit{Negative Log Likelihood} as the loss function to make the boosting procedure available for the unsupervised tasks. From a learning theory viewpoint, we first prove fast convergence rates for GBHT with the smoothness assumption that the underlying density function lies in the space $C^{0,\alpha}$. Then when the target density function lies in spaces $C^{1,\alpha}$, we present an upper bound for GBHT which is smaller than the lower bound of its corresponding base learner, in the sense of convergence rates. To the best of our knowledge, we make the first attempt to theoretically explain why boosting can enhance the performance of its base learners for density estimation problems. In experiments, we not only conduct performance comparisons with the widely used KDE, but also apply GBHT to anomaly detection to showcase a further application of GBHT.} }
Endnote
%0 Conference Paper %T GBHT: Gradient Boosting Histogram Transform for Density Estimation %A Jingyi Cui %A Hanyuan Hang %A Yisen Wang %A Zhouchen Lin %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-cui21c %I PMLR %P 2233--2243 %U https://proceedings.mlr.press/v139/cui21c.html %V 139 %X In this paper, we propose a density estimation algorithm called \textit{Gradient Boosting Histogram Transform} (GBHT), where we adopt the \textit{Negative Log Likelihood} as the loss function to make the boosting procedure available for the unsupervised tasks. From a learning theory viewpoint, we first prove fast convergence rates for GBHT with the smoothness assumption that the underlying density function lies in the space $C^{0,\alpha}$. Then when the target density function lies in spaces $C^{1,\alpha}$, we present an upper bound for GBHT which is smaller than the lower bound of its corresponding base learner, in the sense of convergence rates. To the best of our knowledge, we make the first attempt to theoretically explain why boosting can enhance the performance of its base learners for density estimation problems. In experiments, we not only conduct performance comparisons with the widely used KDE, but also apply GBHT to anomaly detection to showcase a further application of GBHT.
APA
Cui, J., Hang, H., Wang, Y. & Lin, Z.. (2021). GBHT: Gradient Boosting Histogram Transform for Density Estimation. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2233-2243 Available from https://proceedings.mlr.press/v139/cui21c.html.

Related Material