Sample Efficient Learning of Predictors that Complement Humans

Mohammad-Amin Charusaie, Hussein Mozannar, David Sontag, Samira Samadi
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:2972-3005, 2022.

Abstract

One of the goals of learning algorithms is to complement and reduce the burden on human decision makers. The expert deferral setting wherein an algorithm can either predict on its own or defer the decision to a downstream expert helps accomplish this goal. A fundamental aspect of this setting is the need to learn complementary predictors that improve on the human’s weaknesses rather than learning predictors optimized for average error. In this work, we provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. To enable efficiently learning such predictors, we consider a family of consistent surrogate loss functions for expert deferral and analyze their theoretical properties. Finally, we design active learning schemes that require minimal amount of data of human expert predictions in order to learn accurate deferral systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-charusaie22a, title = {Sample Efficient Learning of Predictors that Complement Humans}, author = {Charusaie, Mohammad-Amin and Mozannar, Hussein and Sontag, David and Samadi, Samira}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {2972--3005}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/charusaie22a/charusaie22a.pdf}, url = {https://proceedings.mlr.press/v162/charusaie22a.html}, abstract = {One of the goals of learning algorithms is to complement and reduce the burden on human decision makers. The expert deferral setting wherein an algorithm can either predict on its own or defer the decision to a downstream expert helps accomplish this goal. A fundamental aspect of this setting is the need to learn complementary predictors that improve on the human’s weaknesses rather than learning predictors optimized for average error. In this work, we provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. To enable efficiently learning such predictors, we consider a family of consistent surrogate loss functions for expert deferral and analyze their theoretical properties. Finally, we design active learning schemes that require minimal amount of data of human expert predictions in order to learn accurate deferral systems.} }
Endnote
%0 Conference Paper %T Sample Efficient Learning of Predictors that Complement Humans %A Mohammad-Amin Charusaie %A Hussein Mozannar %A David Sontag %A Samira Samadi %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-charusaie22a %I PMLR %P 2972--3005 %U https://proceedings.mlr.press/v162/charusaie22a.html %V 162 %X One of the goals of learning algorithms is to complement and reduce the burden on human decision makers. The expert deferral setting wherein an algorithm can either predict on its own or defer the decision to a downstream expert helps accomplish this goal. A fundamental aspect of this setting is the need to learn complementary predictors that improve on the human’s weaknesses rather than learning predictors optimized for average error. In this work, we provide the first theoretical analysis of the benefit of learning complementary predictors in expert deferral. To enable efficiently learning such predictors, we consider a family of consistent surrogate loss functions for expert deferral and analyze their theoretical properties. Finally, we design active learning schemes that require minimal amount of data of human expert predictions in order to learn accurate deferral systems.
APA
Charusaie, M., Mozannar, H., Sontag, D. & Samadi, S.. (2022). Sample Efficient Learning of Predictors that Complement Humans. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:2972-3005 Available from https://proceedings.mlr.press/v162/charusaie22a.html.

Related Material