Semi-supervised Learning by Modeling Multiple-Annotator Expertise

Yan Yan, Romer Rosales, Glenn Fung, Jennifer Dy
Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, PMLR R8:673-681, 2010.

Abstract

Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration, multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumstances groundtruth may not exist). Semi-supervised learning approaches have shown that utilizing the unlabeled data is often beneficial in these cases. This paper presents a probabilistic semi-supervised model and algorithm that allows for learning from both unlabeled and labeled data in the presence of multiple annotators. We assume that it is known what annotator labeled which data points. The proposed approach produces annotator models that allow us to provide (1) estimates of the true label and (2) annotator variable expertise for both labeled and unlabeled data. We provide numerical comparisons under various scenarios and with respect to standard semi-supervised learning. Experiments showed that the presented approach provides clear advantages over multi-annotator methods that do not use the unlabeled data and over methods that do not use multi-labeler information.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR8-yan10a, title = {Semi-supervised Learning by Modeling Multiple-Annotator Expertise}, author = {Yan, Yan and Rosales, Romer and Fung, Glenn and Dy, Jennifer}, booktitle = {Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence}, pages = {673--681}, year = {2010}, editor = {Grünwald, Peter and Spirtes, Peter}, volume = {R8}, series = {Proceedings of Machine Learning Research}, month = {08--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r8/main/assets/yan10a/yan10a.pdf}, url = {https://proceedings.mlr.press/r8/yan10a.html}, abstract = {Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration, multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumstances groundtruth may not exist). Semi-supervised learning approaches have shown that utilizing the unlabeled data is often beneficial in these cases. This paper presents a probabilistic semi-supervised model and algorithm that allows for learning from both unlabeled and labeled data in the presence of multiple annotators. We assume that it is known what annotator labeled which data points. The proposed approach produces annotator models that allow us to provide (1) estimates of the true label and (2) annotator variable expertise for both labeled and unlabeled data. We provide numerical comparisons under various scenarios and with respect to standard semi-supervised learning. Experiments showed that the presented approach provides clear advantages over multi-annotator methods that do not use the unlabeled data and over methods that do not use multi-labeler information.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Semi-supervised Learning by Modeling Multiple-Annotator Expertise %A Yan Yan %A Romer Rosales %A Glenn Fung %A Jennifer Dy %B Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2010 %E Peter Grünwald %E Peter Spirtes %F pmlr-vR8-yan10a %I PMLR %P 673--681 %U https://proceedings.mlr.press/r8/yan10a.html %V R8 %X Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration, multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumstances groundtruth may not exist). Semi-supervised learning approaches have shown that utilizing the unlabeled data is often beneficial in these cases. This paper presents a probabilistic semi-supervised model and algorithm that allows for learning from both unlabeled and labeled data in the presence of multiple annotators. We assume that it is known what annotator labeled which data points. The proposed approach produces annotator models that allow us to provide (1) estimates of the true label and (2) annotator variable expertise for both labeled and unlabeled data. We provide numerical comparisons under various scenarios and with respect to standard semi-supervised learning. Experiments showed that the presented approach provides clear advantages over multi-annotator methods that do not use the unlabeled data and over methods that do not use multi-labeler information. %Z Reissued by PMLR on 04 October 2026.
APA
Yan, Y., Rosales, R., Fung, G. & Dy, J.. (2010). Semi-supervised Learning by Modeling Multiple-Annotator Expertise. Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R8:673-681 Available from https://proceedings.mlr.press/r8/yan10a.html. Reissued by PMLR on 04 October 2026.

Related Material