The Label Complexity of Mixed-Initiative Classifier Training

Jina Suh, Xiaojin Zhu, Saleema Amershi
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:2800-2809, 2016.

Abstract

Mixed-initiative classifier training, where the human teacher can choose which items to label or to label items chosen by the computer, has enjoyed empirical success but without a rigorous statistical learning theoretical justification. We analyze the label complexity of a simple mixed-initiative training mechanism using teach- ing dimension and active learning. We show that mixed-initiative training is advantageous com- pared to either computer-initiated (represented by active learning) or human-initiated classifier training. The advantage exists across all human teaching abilities, from optimal to completely unhelpful teachers. We further improve classifier training by educating the human teachers. This is done by showing, or explaining, optimal teaching sets to the human teachers. We conduct Mechanical Turk human experiments on two stylistic classifier training tasks to illustrate our approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-suh16, title = {The Label Complexity of Mixed-Initiative Classifier Training}, author = {Suh, Jina and Zhu, Xiaojin and Amershi, Saleema}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {2800--2809}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/suh16.pdf}, url = {https://proceedings.mlr.press/v48/suh16.html}, abstract = {Mixed-initiative classifier training, where the human teacher can choose which items to label or to label items chosen by the computer, has enjoyed empirical success but without a rigorous statistical learning theoretical justification. We analyze the label complexity of a simple mixed-initiative training mechanism using teach- ing dimension and active learning. We show that mixed-initiative training is advantageous com- pared to either computer-initiated (represented by active learning) or human-initiated classifier training. The advantage exists across all human teaching abilities, from optimal to completely unhelpful teachers. We further improve classifier training by educating the human teachers. This is done by showing, or explaining, optimal teaching sets to the human teachers. We conduct Mechanical Turk human experiments on two stylistic classifier training tasks to illustrate our approach.} }
Endnote
%0 Conference Paper %T The Label Complexity of Mixed-Initiative Classifier Training %A Jina Suh %A Xiaojin Zhu %A Saleema Amershi %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-suh16 %I PMLR %P 2800--2809 %U https://proceedings.mlr.press/v48/suh16.html %V 48 %X Mixed-initiative classifier training, where the human teacher can choose which items to label or to label items chosen by the computer, has enjoyed empirical success but without a rigorous statistical learning theoretical justification. We analyze the label complexity of a simple mixed-initiative training mechanism using teach- ing dimension and active learning. We show that mixed-initiative training is advantageous com- pared to either computer-initiated (represented by active learning) or human-initiated classifier training. The advantage exists across all human teaching abilities, from optimal to completely unhelpful teachers. We further improve classifier training by educating the human teachers. This is done by showing, or explaining, optimal teaching sets to the human teachers. We conduct Mechanical Turk human experiments on two stylistic classifier training tasks to illustrate our approach.
RIS
TY - CPAPER TI - The Label Complexity of Mixed-Initiative Classifier Training AU - Jina Suh AU - Xiaojin Zhu AU - Saleema Amershi BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-suh16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 2800 EP - 2809 L1 - http://proceedings.mlr.press/v48/suh16.pdf UR - https://proceedings.mlr.press/v48/suh16.html AB - Mixed-initiative classifier training, where the human teacher can choose which items to label or to label items chosen by the computer, has enjoyed empirical success but without a rigorous statistical learning theoretical justification. We analyze the label complexity of a simple mixed-initiative training mechanism using teach- ing dimension and active learning. We show that mixed-initiative training is advantageous com- pared to either computer-initiated (represented by active learning) or human-initiated classifier training. The advantage exists across all human teaching abilities, from optimal to completely unhelpful teachers. We further improve classifier training by educating the human teachers. This is done by showing, or explaining, optimal teaching sets to the human teachers. We conduct Mechanical Turk human experiments on two stylistic classifier training tasks to illustrate our approach. ER -
APA
Suh, J., Zhu, X. & Amershi, S.. (2016). The Label Complexity of Mixed-Initiative Classifier Training. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:2800-2809 Available from https://proceedings.mlr.press/v48/suh16.html.

Related Material