Bias-Variance Decompositions for Margin Losses

Danny Wood, Tingting Mu, Gavin Brown
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:1975-2001, 2022.

Abstract

We introduce a novel bias-variance decomposition for a range of strictly convex margin losses, including the logistic loss (minimized by the classic LogitBoost algorithm) as well as the squared margin loss and canonical boosting loss. Furthermore we show that, for all strictly convex margin losses, the expected risk decomposes into the risk of a "central" model and a term quantifying variation in the functional margin with respect to variations in the training data. These decompositions provide a diagnostic tool for practitioners to understand model overfitting/underfitting, and have implications for additive ensemble models—for example, when our bias-variance decomposition holds, there is a corresponding "ambiguity" decomposition, which can be used to quantify model diversity.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-wood22a, title = { Bias-Variance Decompositions for Margin Losses }, author = {Wood, Danny and Mu, Tingting and Brown, Gavin}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {1975--2001}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/wood22a/wood22a.pdf}, url = {https://proceedings.mlr.press/v151/wood22a.html}, abstract = { We introduce a novel bias-variance decomposition for a range of strictly convex margin losses, including the logistic loss (minimized by the classic LogitBoost algorithm) as well as the squared margin loss and canonical boosting loss. Furthermore we show that, for all strictly convex margin losses, the expected risk decomposes into the risk of a "central" model and a term quantifying variation in the functional margin with respect to variations in the training data. These decompositions provide a diagnostic tool for practitioners to understand model overfitting/underfitting, and have implications for additive ensemble models—for example, when our bias-variance decomposition holds, there is a corresponding "ambiguity" decomposition, which can be used to quantify model diversity. } }
Endnote
%0 Conference Paper %T Bias-Variance Decompositions for Margin Losses %A Danny Wood %A Tingting Mu %A Gavin Brown %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-wood22a %I PMLR %P 1975--2001 %U https://proceedings.mlr.press/v151/wood22a.html %V 151 %X We introduce a novel bias-variance decomposition for a range of strictly convex margin losses, including the logistic loss (minimized by the classic LogitBoost algorithm) as well as the squared margin loss and canonical boosting loss. Furthermore we show that, for all strictly convex margin losses, the expected risk decomposes into the risk of a "central" model and a term quantifying variation in the functional margin with respect to variations in the training data. These decompositions provide a diagnostic tool for practitioners to understand model overfitting/underfitting, and have implications for additive ensemble models—for example, when our bias-variance decomposition holds, there is a corresponding "ambiguity" decomposition, which can be used to quantify model diversity.
APA
Wood, D., Mu, T. & Brown, G.. (2022). Bias-Variance Decompositions for Margin Losses . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:1975-2001 Available from https://proceedings.mlr.press/v151/wood22a.html.

Related Material