Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, Yixuan Li
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:36868-36886, 2023.

Abstract

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-wei23e, title = {Mitigating Memorization of Noisy Labels by Clipping the Model Prediction}, author = {Wei, Hongxin and Zhuang, Huiping and Xie, Renchunzi and Feng, Lei and Niu, Gang and An, Bo and Li, Yixuan}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {36868--36886}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/wei23e/wei23e.pdf}, url = {https://proceedings.mlr.press/v202/wei23e.html}, abstract = {In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.} }
Endnote
%0 Conference Paper %T Mitigating Memorization of Noisy Labels by Clipping the Model Prediction %A Hongxin Wei %A Huiping Zhuang %A Renchunzi Xie %A Lei Feng %A Gang Niu %A Bo An %A Yixuan Li %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-wei23e %I PMLR %P 36868--36886 %U https://proceedings.mlr.press/v202/wei23e.html %V 202 %X In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.
APA
Wei, H., Zhuang, H., Xie, R., Feng, L., Niu, G., An, B. & Li, Y.. (2023). Mitigating Memorization of Noisy Labels by Clipping the Model Prediction. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:36868-36886 Available from https://proceedings.mlr.press/v202/wei23e.html.

Related Material