Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing

Yuan Jin, Mark Carman, Ye Zhu, Wray Buntine
Proceedings of The 10th Asian Conference on Machine Learning, PMLR 95:192-207, 2018.

Abstract

The questions in a crowdsourcing task typically exhibit varying degrees of difficulty and subjectivity. Their joint effects give rise to the variation in responses to the same question by different crowd-workers. This variation is low when the question is easy to answer and objective, and high when it is difficult and subjective. Unfortunately, current quality control methods for crowdsourcing consider only the question difficulty to account for the variation. As a result, these methods cannot distinguish workers' ,personal preferences for different correct answers of a partially subjective question from their ability to avoid objectively incorrect answers for that question. To address this issue, we present a probabilistic model which (i) explicitly encodes question difficulty as a model parameter and (ii) implicitly encodes question subjectivity via latent preference factors for crowd-workers. We show that question subjectivity induces grouping of crowd-workers, revealed through clustering of their latent preferences. Moreover, we develop a quantitative measure for the question subjectivity. Experiments show that our model (1) improves both the question true answer prediction and the unseen worker response prediction, and (2) can potentially provide rankings of questions coherent with human assessment in terms of difficulty and subjectivity.

Cite this Paper

BibTeX
@InProceedings{pmlr-v95-jin18a, title = {Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing}, author = {Jin, Yuan and Carman, Mark and Zhu, Ye and Buntine, Wray}, booktitle = {Proceedings of The 10th Asian Conference on Machine Learning}, pages = {192--207}, year = {2018}, editor = {Zhu, Jun and Takeuchi, Ichiro}, volume = {95}, series = {Proceedings of Machine Learning Research}, month = {14--16 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v95/jin18a/jin18a.pdf}, url = {https://proceedings.mlr.press/v95/jin18a.html}, abstract = {The questions in a crowdsourcing task typically exhibit varying degrees of difficulty and subjectivity. Their joint effects give rise to the variation in responses to the same question by different crowd-workers. This variation is low when the question is easy to answer and objective, and high when it is difficult and subjective. Unfortunately, current quality control methods for crowdsourcing consider only the question difficulty to account for the variation. As a result, these methods cannot distinguish workers' ,personal preferences for different correct answers of a partially subjective question from their ability to avoid objectively incorrect answers for that question. To address this issue, we present a probabilistic model which (i) explicitly encodes question difficulty as a model parameter and (ii) implicitly encodes question subjectivity via latent preference factors for crowd-workers. We show that question subjectivity induces grouping of crowd-workers, revealed through clustering of their latent preferences. Moreover, we develop a quantitative measure for the question subjectivity. Experiments show that our model (1) improves both the question true answer prediction and the unseen worker response prediction, and (2) can potentially provide rankings of questions coherent with human assessment in terms of difficulty and subjectivity.} }
Endnote
%0 Conference Paper %T Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing %A Yuan Jin %A Mark Carman %A Ye Zhu %A Wray Buntine %B Proceedings of The 10th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jun Zhu %E Ichiro Takeuchi %F pmlr-v95-jin18a %I PMLR %P 192--207 %U https://proceedings.mlr.press/v95/jin18a.html %V 95 %X The questions in a crowdsourcing task typically exhibit varying degrees of difficulty and subjectivity. Their joint effects give rise to the variation in responses to the same question by different crowd-workers. This variation is low when the question is easy to answer and objective, and high when it is difficult and subjective. Unfortunately, current quality control methods for crowdsourcing consider only the question difficulty to account for the variation. As a result, these methods cannot distinguish workers' ,personal preferences for different correct answers of a partially subjective question from their ability to avoid objectively incorrect answers for that question. To address this issue, we present a probabilistic model which (i) explicitly encodes question difficulty as a model parameter and (ii) implicitly encodes question subjectivity via latent preference factors for crowd-workers. We show that question subjectivity induces grouping of crowd-workers, revealed through clustering of their latent preferences. Moreover, we develop a quantitative measure for the question subjectivity. Experiments show that our model (1) improves both the question true answer prediction and the unseen worker response prediction, and (2) can potentially provide rankings of questions coherent with human assessment in terms of difficulty and subjectivity.
APA
Jin, Y., Carman, M., Zhu, Y. & Buntine, W.. (2018). Distinguishing Question Subjectivity from Difficulty for Improved Crowdsourcing. Proceedings of The 10th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 95:192-207 Available from https://proceedings.mlr.press/v95/jin18a.html.

Related Material