Consistent Estimators for Learning to Defer to an Expert

Hussein Mozannar, David Sontag
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7076-7087, 2020.

Abstract

Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert’s decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-mozannar20b, title = {Consistent Estimators for Learning to Defer to an Expert}, author = {Mozannar, Hussein and Sontag, David}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {7076--7087}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/mozannar20b/mozannar20b.pdf}, url = {https://proceedings.mlr.press/v119/mozannar20b.html}, abstract = {Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert’s decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.} }
Endnote
%0 Conference Paper %T Consistent Estimators for Learning to Defer to an Expert %A Hussein Mozannar %A David Sontag %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-mozannar20b %I PMLR %P 7076--7087 %U https://proceedings.mlr.press/v119/mozannar20b.html %V 119 %X Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert’s decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.
APA
Mozannar, H. & Sontag, D.. (2020). Consistent Estimators for Learning to Defer to an Expert. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:7076-7087 Available from https://proceedings.mlr.press/v119/mozannar20b.html.

Related Material