Smoothly Giving up: Robustness for Simple Models

Tyler Sypherd, Nathaniel Stromberg, Richard Nock, Visar Berisha, Lalitha Sankar
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:5376-5410, 2023.

Abstract

There is a growing need for models that are interpretable and have reduced energy/computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $\alpha$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $\alpha$ hyperparameter smoothly introduces non-convexity and offers the benefit of “giving up” on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-sypherd23a, title = {Smoothly Giving up: Robustness for Simple Models}, author = {Sypherd, Tyler and Stromberg, Nathaniel and Nock, Richard and Berisha, Visar and Sankar, Lalitha}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {5376--5410}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/sypherd23a/sypherd23a.pdf}, url = {https://proceedings.mlr.press/v206/sypherd23a.html}, abstract = {There is a growing need for models that are interpretable and have reduced energy/computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $\alpha$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $\alpha$ hyperparameter smoothly introduces non-convexity and offers the benefit of “giving up” on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.} }
Endnote
%0 Conference Paper %T Smoothly Giving up: Robustness for Simple Models %A Tyler Sypherd %A Nathaniel Stromberg %A Richard Nock %A Visar Berisha %A Lalitha Sankar %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-sypherd23a %I PMLR %P 5376--5410 %U https://proceedings.mlr.press/v206/sypherd23a.html %V 206 %X There is a growing need for models that are interpretable and have reduced energy/computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based $\alpha$-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the $\alpha$ hyperparameter smoothly introduces non-convexity and offers the benefit of “giving up” on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.
APA
Sypherd, T., Stromberg, N., Nock, R., Berisha, V. & Sankar, L.. (2023). Smoothly Giving up: Robustness for Simple Models. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:5376-5410 Available from https://proceedings.mlr.press/v206/sypherd23a.html.

Related Material