Unsupervised Label Noise Modeling and Loss Correction

Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, Kevin Mcguinness
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:312-321, 2019.

Abstract

Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-arazo19a, title = {Unsupervised Label Noise Modeling and Loss Correction}, author = {Arazo, Eric and Ortego, Diego and Albert, Paul and O'Connor, Noel and Mcguinness, Kevin}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {312--321}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/arazo19a/arazo19a.pdf}, url = {https://proceedings.mlr.press/v97/arazo19a.html}, abstract = {Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.} }
Endnote
%0 Conference Paper %T Unsupervised Label Noise Modeling and Loss Correction %A Eric Arazo %A Diego Ortego %A Paul Albert %A Noel O’Connor %A Kevin Mcguinness %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-arazo19a %I PMLR %P 312--321 %U https://proceedings.mlr.press/v97/arazo19a.html %V 97 %X Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.
APA
Arazo, E., Ortego, D., Albert, P., O’Connor, N. & Mcguinness, K.. (2019). Unsupervised Label Noise Modeling and Loss Correction. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:312-321 Available from https://proceedings.mlr.press/v97/arazo19a.html.

Related Material