Semi-Automated Data Labeling

Michael Desmond, Evelyn Duesterwald, Kristina Brimijoin, Michelle Brachman, Qian Pan
Proceedings of the NeurIPS 2020 Competition and Demonstration Track, PMLR 133:156-169, 2021.

Abstract

Labeling data is often a tedious and error-prone activity. However, organizing the labeling experience as a human-machine collaboration has the potential to improve label quality and reduce human effort. In this paper we describe a semi-automated data labeling system which employs a predictive model to guide and assist the human labeler. The model learns by observing labeling decisions, and is used to recommend labels and automate basic functions in the labeling interface. Agreement between the labeler and the model is tracked and presented via a system of checkpoints. At each checkpoint the labeler has the opportunity to delegate the remainder of the labeling task to the model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v133-desmond21a, title = {Semi-Automated Data Labeling}, author = {Desmond, Michael and Duesterwald, Evelyn and Brimijoin, Kristina and Brachman, Michelle and Pan, Qian}, booktitle = {Proceedings of the NeurIPS 2020 Competition and Demonstration Track}, pages = {156--169}, year = {2021}, editor = {Escalante, Hugo Jair and Hofmann, Katja}, volume = {133}, series = {Proceedings of Machine Learning Research}, month = {06--12 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v133/desmond21a/desmond21a.pdf}, url = {https://proceedings.mlr.press/v133/desmond21a.html}, abstract = {Labeling data is often a tedious and error-prone activity. However, organizing the labeling experience as a human-machine collaboration has the potential to improve label quality and reduce human effort. In this paper we describe a semi-automated data labeling system which employs a predictive model to guide and assist the human labeler. The model learns by observing labeling decisions, and is used to recommend labels and automate basic functions in the labeling interface. Agreement between the labeler and the model is tracked and presented via a system of checkpoints. At each checkpoint the labeler has the opportunity to delegate the remainder of the labeling task to the model. } }
Endnote
%0 Conference Paper %T Semi-Automated Data Labeling %A Michael Desmond %A Evelyn Duesterwald %A Kristina Brimijoin %A Michelle Brachman %A Qian Pan %B Proceedings of the NeurIPS 2020 Competition and Demonstration Track %C Proceedings of Machine Learning Research %D 2021 %E Hugo Jair Escalante %E Katja Hofmann %F pmlr-v133-desmond21a %I PMLR %P 156--169 %U https://proceedings.mlr.press/v133/desmond21a.html %V 133 %X Labeling data is often a tedious and error-prone activity. However, organizing the labeling experience as a human-machine collaboration has the potential to improve label quality and reduce human effort. In this paper we describe a semi-automated data labeling system which employs a predictive model to guide and assist the human labeler. The model learns by observing labeling decisions, and is used to recommend labels and automate basic functions in the labeling interface. Agreement between the labeler and the model is tracked and presented via a system of checkpoints. At each checkpoint the labeler has the opportunity to delegate the remainder of the labeling task to the model.
APA
Desmond, M., Duesterwald, E., Brimijoin, K., Brachman, M. & Pan, Q.. (2021). Semi-Automated Data Labeling. Proceedings of the NeurIPS 2020 Competition and Demonstration Track, in Proceedings of Machine Learning Research 133:156-169 Available from https://proceedings.mlr.press/v133/desmond21a.html.

Related Material