Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy

Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):262-270, 2014.

Abstract

We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-zhouc14, title = {Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy}, author = {Zhou, Dengyong and Liu, Qiang and Platt, John and Meek, Christopher}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {262--270}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/zhouc14.pdf}, url = {https://proceedings.mlr.press/v32/zhouc14.html}, abstract = {We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.} }
Endnote
%0 Conference Paper %T Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy %A Dengyong Zhou %A Qiang Liu %A John Platt %A Christopher Meek %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-zhouc14 %I PMLR %P 262--270 %U https://proceedings.mlr.press/v32/zhouc14.html %V 32 %N 2 %X We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.
RIS
TY - CPAPER TI - Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy AU - Dengyong Zhou AU - Qiang Liu AU - John Platt AU - Christopher Meek BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-zhouc14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 262 EP - 270 L1 - http://proceedings.mlr.press/v32/zhouc14.pdf UR - https://proceedings.mlr.press/v32/zhouc14.html AB - We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods. ER -
APA
Zhou, D., Liu, Q., Platt, J. & Meek, C.. (2014). Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):262-270 Available from https://proceedings.mlr.press/v32/zhouc14.html.

Related Material