Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy

Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek
; Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):262-270, 2014.

Abstract

We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.

Cite this Paper


BibTeX
@InProceedings{pmlr-v32-zhouc14, title = {Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy}, author = {Dengyong Zhou and Qiang Liu and John Platt and Christopher Meek}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {262--270}, year = {2014}, editor = {Eric P. Xing and Tony Jebara}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/zhouc14.pdf}, url = {http://proceedings.mlr.press/v32/zhouc14.html}, abstract = {We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.} }
Endnote
%0 Conference Paper %T Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy %A Dengyong Zhou %A Qiang Liu %A John Platt %A Christopher Meek %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-zhouc14 %I PMLR %J Proceedings of Machine Learning Research %P 262--270 %U http://proceedings.mlr.press %V 32 %N 2 %W PMLR %X We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.
RIS
TY - CPAPER TI - Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy AU - Dengyong Zhou AU - Qiang Liu AU - John Platt AU - Christopher Meek BT - Proceedings of the 31st International Conference on Machine Learning PY - 2014/01/27 DA - 2014/01/27 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-zhouc14 PB - PMLR SP - 262 DP - PMLR EP - 270 L1 - http://proceedings.mlr.press/v32/zhouc14.pdf UR - http://proceedings.mlr.press/v32/zhouc14.html AB - We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods. ER -
APA
Zhou, D., Liu, Q., Platt, J. & Meek, C.. (2014). Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy. Proceedings of the 31st International Conference on Machine Learning, in PMLR 32(2):262-270

Related Material