Aggregating Ordinal Labels from Crowds by Minimax Conditional Entropy


Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek ;
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):262-270, 2014.


We propose a method to aggregate noisy ordinal labels collected from a crowd of workers or annotators. Eliciting ordinal labels is important in tasks such as judging web search quality and consumer satisfaction. Our method is motivated by the observation that workers usually have difficulty distinguishing between two adjacent ordinal classes whereas distinguishing between two classes which are far away from each other is much easier. We develop the method through minimax conditional entropy subject to constraints which encode this observation. Empirical evaluations on real datasets demonstrate significant improvements over existing methods.

Related Material