Gaussian Process Classification and Active Learning with Multiple Annotators

Filipe Rodrigues, Francisco Pereira, Bernardete Ribeiro
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):433-441, 2014.

Abstract

Learning from multiple annotators took a valuable step towards modelling data that does not fit the usual single annotator setting. However, multiple annotators sometimes offer varying degrees of expertise. When disagreements arise, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we extend GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework to build proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-rodrigues14, title = {Gaussian Process Classification and Active Learning with Multiple Annotators}, author = {Rodrigues, Filipe and Pereira, Francisco and Ribeiro, Bernardete}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {433--441}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/rodrigues14.pdf}, url = {https://proceedings.mlr.press/v32/rodrigues14.html}, abstract = {Learning from multiple annotators took a valuable step towards modelling data that does not fit the usual single annotator setting. However, multiple annotators sometimes offer varying degrees of expertise. When disagreements arise, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we extend GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework to build proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.} }
Endnote
%0 Conference Paper %T Gaussian Process Classification and Active Learning with Multiple Annotators %A Filipe Rodrigues %A Francisco Pereira %A Bernardete Ribeiro %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-rodrigues14 %I PMLR %P 433--441 %U https://proceedings.mlr.press/v32/rodrigues14.html %V 32 %N 2 %X Learning from multiple annotators took a valuable step towards modelling data that does not fit the usual single annotator setting. However, multiple annotators sometimes offer varying degrees of expertise. When disagreements arise, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we extend GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework to build proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.
RIS
TY - CPAPER TI - Gaussian Process Classification and Active Learning with Multiple Annotators AU - Filipe Rodrigues AU - Francisco Pereira AU - Bernardete Ribeiro BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-rodrigues14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 433 EP - 441 L1 - http://proceedings.mlr.press/v32/rodrigues14.pdf UR - https://proceedings.mlr.press/v32/rodrigues14.html AB - Learning from multiple annotators took a valuable step towards modelling data that does not fit the usual single annotator setting. However, multiple annotators sometimes offer varying degrees of expertise. When disagreements arise, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we extend GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework to build proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further. ER -
APA
Rodrigues, F., Pereira, F. & Ribeiro, B.. (2014). Gaussian Process Classification and Active Learning with Multiple Annotators. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):433-441 Available from https://proceedings.mlr.press/v32/rodrigues14.html.

Related Material