Transfer and Marginalize: Explaining Away Label Noise with Privileged Information

Mark Collier, Rodolphe Jenatton, Effrosyni Kokiopoulou, Jesse Berent
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4219-4237, 2022.

Abstract

Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop a simple and efficient method for supervised learning with neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test-time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-collier22a, title = {Transfer and Marginalize: Explaining Away Label Noise with Privileged Information}, author = {Collier, Mark and Jenatton, Rodolphe and Kokiopoulou, Effrosyni and Berent, Jesse}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {4219--4237}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/collier22a/collier22a.pdf}, url = {https://proceedings.mlr.press/v162/collier22a.html}, abstract = {Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop a simple and efficient method for supervised learning with neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test-time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.} }
Endnote
%0 Conference Paper %T Transfer and Marginalize: Explaining Away Label Noise with Privileged Information %A Mark Collier %A Rodolphe Jenatton %A Effrosyni Kokiopoulou %A Jesse Berent %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-collier22a %I PMLR %P 4219--4237 %U https://proceedings.mlr.press/v162/collier22a.html %V 162 %X Supervised learning datasets often have privileged information, in the form of features which are available at training time but are not available at test time e.g. the ID of the annotator that provided the label. We argue that privileged information is useful for explaining away label noise, thereby reducing the harmful impact of noisy labels. We develop a simple and efficient method for supervised learning with neural networks: it transfers via weight sharing the knowledge learned with privileged information and approximately marginalizes over privileged information at test time. Our method, TRAM (TRansfer and Marginalize), has minimal training time overhead and has the same test-time cost as not using privileged information. TRAM performs strongly on CIFAR-10H, ImageNet and Civil Comments benchmarks.
APA
Collier, M., Jenatton, R., Kokiopoulou, E. & Berent, J.. (2022). Transfer and Marginalize: Explaining Away Label Noise with Privileged Information. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:4219-4237 Available from https://proceedings.mlr.press/v162/collier22a.html.

Related Material