Detecting Noisy Labels Using Early Stopped Models

Lea Hergert, Mark Jelasity
Proceedings of The Workshop on Classifier Learning from Difficult Data, PMLR 263:40-47, 2024.

Abstract

We are concerned with the problem of identifying samples with noisy labels in a given dataset. Using the predictions of a well-generalizing model to flag incorrectly predicted labels as noisy is a known method but it is not considered competitive. At the same time, it has been observed recently that gradient descent fits clean samples first, and the noisy samples are memorized later. Inspired by related theoretical results, we revisit the idea of using the predictions of an early stopped model to classify samples as noisy or clean. We offer two key improvements that allow this strikingly simple approach to outperform some well-known methods. First, we use the model over its own training set to directly exploit the so-called clean priority learning phenomenon. Second, we use an ensemble of model check points around the early stopping point to reduce the variance of the predictions. We also introduce a novel method that makes use of the same early stopped model ensemble, but classifies samples based on the per-sample gradient of the loss, motivated by recent theoretical results on clean priority learning. Our approaches only passively observe a normal training run and collect checkpoints. No extra input samples are added, no thresholds are tuned, and no pre-trained models are used. Through empirical evaluations, we demonstrate that our methods are competitive with other approaches from related work for both detecting noisy samples and for noise-filtered training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v263-lea24a, title = {Detecting Noisy Labels Using Early Stopped Models}, author = {Hergert, Lea and Jelasity, Mark}, booktitle = {Proceedings of The Workshop on Classifier Learning from Difficult Data}, pages = {40--47}, year = {2024}, editor = {Zyblewski, Pawel and Grana, Manuel and Pawel, Ksieniewicz and Minku, Leandro}, volume = {263}, series = {Proceedings of Machine Learning Research}, month = {19--20 Oct}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v263/main/assets/lea24a/lea24a.pdf}, url = {https://proceedings.mlr.press/v263/lea24a.html}, abstract = {We are concerned with the problem of identifying samples with noisy labels in a given dataset. Using the predictions of a well-generalizing model to flag incorrectly predicted labels as noisy is a known method but it is not considered competitive. At the same time, it has been observed recently that gradient descent fits clean samples first, and the noisy samples are memorized later. Inspired by related theoretical results, we revisit the idea of using the predictions of an early stopped model to classify samples as noisy or clean. We offer two key improvements that allow this strikingly simple approach to outperform some well-known methods. First, we use the model over its own training set to directly exploit the so-called clean priority learning phenomenon. Second, we use an ensemble of model check points around the early stopping point to reduce the variance of the predictions. We also introduce a novel method that makes use of the same early stopped model ensemble, but classifies samples based on the per-sample gradient of the loss, motivated by recent theoretical results on clean priority learning. Our approaches only passively observe a normal training run and collect checkpoints. No extra input samples are added, no thresholds are tuned, and no pre-trained models are used. Through empirical evaluations, we demonstrate that our methods are competitive with other approaches from related work for both detecting noisy samples and for noise-filtered training.} }
Endnote
%0 Conference Paper %T Detecting Noisy Labels Using Early Stopped Models %A Lea Hergert %A Mark Jelasity %B Proceedings of The Workshop on Classifier Learning from Difficult Data %C Proceedings of Machine Learning Research %D 2024 %E Pawel Zyblewski %E Manuel Grana %E Ksieniewicz Pawel %E Leandro Minku %F pmlr-v263-lea24a %I PMLR %P 40--47 %U https://proceedings.mlr.press/v263/lea24a.html %V 263 %X We are concerned with the problem of identifying samples with noisy labels in a given dataset. Using the predictions of a well-generalizing model to flag incorrectly predicted labels as noisy is a known method but it is not considered competitive. At the same time, it has been observed recently that gradient descent fits clean samples first, and the noisy samples are memorized later. Inspired by related theoretical results, we revisit the idea of using the predictions of an early stopped model to classify samples as noisy or clean. We offer two key improvements that allow this strikingly simple approach to outperform some well-known methods. First, we use the model over its own training set to directly exploit the so-called clean priority learning phenomenon. Second, we use an ensemble of model check points around the early stopping point to reduce the variance of the predictions. We also introduce a novel method that makes use of the same early stopped model ensemble, but classifies samples based on the per-sample gradient of the loss, motivated by recent theoretical results on clean priority learning. Our approaches only passively observe a normal training run and collect checkpoints. No extra input samples are added, no thresholds are tuned, and no pre-trained models are used. Through empirical evaluations, we demonstrate that our methods are competitive with other approaches from related work for both detecting noisy samples and for noise-filtered training.
APA
Hergert, L. & Jelasity, M.. (2024). Detecting Noisy Labels Using Early Stopped Models. Proceedings of The Workshop on Classifier Learning from Difficult Data, in Proceedings of Machine Learning Research 263:40-47 Available from https://proceedings.mlr.press/v263/lea24a.html.

Related Material