Boosting with the Logistic Loss is Consistent

Matus Telgarsky
Proceedings of the 26th Annual Conference on Learning Theory, PMLR 30:911-965, 2013.

Abstract

This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side).The heart of the analysis is to show that, in lieu of explicit regularization and constraints, the structure of the problem is fairly rigidly controlled by the source distribution itself. The first control of this type is in the separable case, where a distribution-dependent relaxed weak learning rate induces speedy convergence with high probability over any sample. Otherwise, in the nonseparable case, the convex surrogate risk itself exhibits distribution-dependent levels of curvature, and consequently the algorithm’s output has small norm with high probability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v30-Telgarsky13, title = {Boosting with the Logistic Loss is Consistent}, author = {Telgarsky, Matus}, booktitle = {Proceedings of the 26th Annual Conference on Learning Theory}, pages = {911--965}, year = {2013}, editor = {Shalev-Shwartz, Shai and Steinwart, Ingo}, volume = {30}, series = {Proceedings of Machine Learning Research}, address = {Princeton, NJ, USA}, month = {12--14 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v30/Telgarsky13.pdf}, url = {https://proceedings.mlr.press/v30/Telgarsky13.html}, abstract = {This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side).The heart of the analysis is to show that, in lieu of explicit regularization and constraints, the structure of the problem is fairly rigidly controlled by the source distribution itself. The first control of this type is in the separable case, where a distribution-dependent relaxed weak learning rate induces speedy convergence with high probability over any sample. Otherwise, in the nonseparable case, the convex surrogate risk itself exhibits distribution-dependent levels of curvature, and consequently the algorithm’s output has small norm with high probability.} }
Endnote
%0 Conference Paper %T Boosting with the Logistic Loss is Consistent %A Matus Telgarsky %B Proceedings of the 26th Annual Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2013 %E Shai Shalev-Shwartz %E Ingo Steinwart %F pmlr-v30-Telgarsky13 %I PMLR %P 911--965 %U https://proceedings.mlr.press/v30/Telgarsky13.html %V 30 %X This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side).The heart of the analysis is to show that, in lieu of explicit regularization and constraints, the structure of the problem is fairly rigidly controlled by the source distribution itself. The first control of this type is in the separable case, where a distribution-dependent relaxed weak learning rate induces speedy convergence with high probability over any sample. Otherwise, in the nonseparable case, the convex surrogate risk itself exhibits distribution-dependent levels of curvature, and consequently the algorithm’s output has small norm with high probability.
RIS
TY - CPAPER TI - Boosting with the Logistic Loss is Consistent AU - Matus Telgarsky BT - Proceedings of the 26th Annual Conference on Learning Theory DA - 2013/06/13 ED - Shai Shalev-Shwartz ED - Ingo Steinwart ID - pmlr-v30-Telgarsky13 PB - PMLR DP - Proceedings of Machine Learning Research VL - 30 SP - 911 EP - 965 L1 - http://proceedings.mlr.press/v30/Telgarsky13.pdf UR - https://proceedings.mlr.press/v30/Telgarsky13.html AB - This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side).The heart of the analysis is to show that, in lieu of explicit regularization and constraints, the structure of the problem is fairly rigidly controlled by the source distribution itself. The first control of this type is in the separable case, where a distribution-dependent relaxed weak learning rate induces speedy convergence with high probability over any sample. Otherwise, in the nonseparable case, the convex surrogate risk itself exhibits distribution-dependent levels of curvature, and consequently the algorithm’s output has small norm with high probability. ER -
APA
Telgarsky, M.. (2013). Boosting with the Logistic Loss is Consistent. Proceedings of the 26th Annual Conference on Learning Theory, in Proceedings of Machine Learning Research 30:911-965 Available from https://proceedings.mlr.press/v30/Telgarsky13.html.

Related Material