Does label smoothing mitigate label noise?

Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, Sanjiv Kumar
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:6448-6458, 2020.

Abstract

Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem — being equivalent to injecting symmetric noise to the labels — we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-lukasik20a, title = {Does label smoothing mitigate label noise?}, author = {Lukasik, Michal and Bhojanapalli, Srinadh and Menon, Aditya and Kumar, Sanjiv}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {6448--6458}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/lukasik20a/lukasik20a.pdf}, url = {https://proceedings.mlr.press/v119/lukasik20a.html}, abstract = {Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem — being equivalent to injecting symmetric noise to the labels — we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.} }
Endnote
%0 Conference Paper %T Does label smoothing mitigate label noise? %A Michal Lukasik %A Srinadh Bhojanapalli %A Aditya Menon %A Sanjiv Kumar %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-lukasik20a %I PMLR %P 6448--6458 %U https://proceedings.mlr.press/v119/lukasik20a.html %V 119 %X Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem — being equivalent to injecting symmetric noise to the labels — we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.
APA
Lukasik, M., Bhojanapalli, S., Menon, A. & Kumar, S.. (2020). Does label smoothing mitigate label noise?. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:6448-6458 Available from https://proceedings.mlr.press/v119/lukasik20a.html.

Related Material