Robust Deep Learning from Crowds with Belief Propagation

Hoyoung Kim, Seunghyuk Cho, Dongwoo Kim, Jungseul Ok
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:2803-2822, 2022.

Abstract

Crowdsourcing systems enable us to collect large-scale dataset, but inherently suffer from noisy labels of low-paid workers. We address the inference and learning problems using such a crowdsourced dataset with noise. Due to the nature of sparsity in crowdsourcing, it is critical to exploit both probabilistic model to capture worker prior and neural network to extract task feature despite risks from wrong prior and overfitted feature in practice. We hence establish a neural-powered Bayesian framework, from which we devise deepMF and deepBP with different choice of variational approximation methods, mean field (MF) and belief propagation (BP), respectively. This provides a unified view of existing methods, which are special cases of deepMF with different priors. In addition, our empirical study suggests that deepBP is a new approach, which is more robust against wrong prior, feature overfitting and extreme workers thanks to the more sophisticated BP than MF.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-kim22a, title = { Robust Deep Learning from Crowds with Belief Propagation }, author = {Kim, Hoyoung and Cho, Seunghyuk and Kim, Dongwoo and Ok, Jungseul}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {2803--2822}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/kim22a/kim22a.pdf}, url = {https://proceedings.mlr.press/v151/kim22a.html}, abstract = { Crowdsourcing systems enable us to collect large-scale dataset, but inherently suffer from noisy labels of low-paid workers. We address the inference and learning problems using such a crowdsourced dataset with noise. Due to the nature of sparsity in crowdsourcing, it is critical to exploit both probabilistic model to capture worker prior and neural network to extract task feature despite risks from wrong prior and overfitted feature in practice. We hence establish a neural-powered Bayesian framework, from which we devise deepMF and deepBP with different choice of variational approximation methods, mean field (MF) and belief propagation (BP), respectively. This provides a unified view of existing methods, which are special cases of deepMF with different priors. In addition, our empirical study suggests that deepBP is a new approach, which is more robust against wrong prior, feature overfitting and extreme workers thanks to the more sophisticated BP than MF. } }
Endnote
%0 Conference Paper %T Robust Deep Learning from Crowds with Belief Propagation %A Hoyoung Kim %A Seunghyuk Cho %A Dongwoo Kim %A Jungseul Ok %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-kim22a %I PMLR %P 2803--2822 %U https://proceedings.mlr.press/v151/kim22a.html %V 151 %X Crowdsourcing systems enable us to collect large-scale dataset, but inherently suffer from noisy labels of low-paid workers. We address the inference and learning problems using such a crowdsourced dataset with noise. Due to the nature of sparsity in crowdsourcing, it is critical to exploit both probabilistic model to capture worker prior and neural network to extract task feature despite risks from wrong prior and overfitted feature in practice. We hence establish a neural-powered Bayesian framework, from which we devise deepMF and deepBP with different choice of variational approximation methods, mean field (MF) and belief propagation (BP), respectively. This provides a unified view of existing methods, which are special cases of deepMF with different priors. In addition, our empirical study suggests that deepBP is a new approach, which is more robust against wrong prior, feature overfitting and extreme workers thanks to the more sophisticated BP than MF.
APA
Kim, H., Cho, S., Kim, D. & Ok, J.. (2022). Robust Deep Learning from Crowds with Belief Propagation . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:2803-2822 Available from https://proceedings.mlr.press/v151/kim22a.html.

Related Material