Iterative Bayesian Learning for Crowdsourced Regression

Jungseul Ok, Sewoong Oh, Yunhun Jang, Jinwoo Shin, Yung Yi
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:1486-1495, 2019.

Abstract

Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, a common practice is to add redundancy by assigning multiple workers to each task and then simply average out these answers. However, to fully harness the wisdom of the crowd, one needs to learn the heterogeneous quality of each worker. We resolve this fundamental challenge in crowdsourced regression tasks, i.e., the answer takes continuous labels, where identifying good or bad workers becomes much more non-trivial compared to a classification setting of discrete labels. In particular, we introduce a Bayesian iterative scheme and show that it provably achieves the optimal mean squared error. Our evaluations on synthetic and real-world datasets support our theoretical results and show the superiority of the proposed scheme.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-ok19a, title = {Iterative Bayesian Learning for Crowdsourced Regression}, author = {Ok, Jungseul and Oh, Sewoong and Jang, Yunhun and Shin, Jinwoo and Yi, Yung}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {1486--1495}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/ok19a/ok19a.pdf}, url = {https://proceedings.mlr.press/v89/ok19a.html}, abstract = {Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, a common practice is to add redundancy by assigning multiple workers to each task and then simply average out these answers. However, to fully harness the wisdom of the crowd, one needs to learn the heterogeneous quality of each worker. We resolve this fundamental challenge in crowdsourced regression tasks, i.e., the answer takes continuous labels, where identifying good or bad workers becomes much more non-trivial compared to a classification setting of discrete labels. In particular, we introduce a Bayesian iterative scheme and show that it provably achieves the optimal mean squared error. Our evaluations on synthetic and real-world datasets support our theoretical results and show the superiority of the proposed scheme.} }
Endnote
%0 Conference Paper %T Iterative Bayesian Learning for Crowdsourced Regression %A Jungseul Ok %A Sewoong Oh %A Yunhun Jang %A Jinwoo Shin %A Yung Yi %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-ok19a %I PMLR %P 1486--1495 %U https://proceedings.mlr.press/v89/ok19a.html %V 89 %X Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, a common practice is to add redundancy by assigning multiple workers to each task and then simply average out these answers. However, to fully harness the wisdom of the crowd, one needs to learn the heterogeneous quality of each worker. We resolve this fundamental challenge in crowdsourced regression tasks, i.e., the answer takes continuous labels, where identifying good or bad workers becomes much more non-trivial compared to a classification setting of discrete labels. In particular, we introduce a Bayesian iterative scheme and show that it provably achieves the optimal mean squared error. Our evaluations on synthetic and real-world datasets support our theoretical results and show the superiority of the proposed scheme.
APA
Ok, J., Oh, S., Jang, Y., Shin, J. & Yi, Y.. (2019). Iterative Bayesian Learning for Crowdsourced Regression. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:1486-1495 Available from https://proceedings.mlr.press/v89/ok19a.html.

Related Material