Performance Metric Elicitation from Pairwise Classifier Comparisons

Gaurush Hiranandani, Shant Boodaghians, Ruta Mehta, Oluwasanmi Koyejo
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:371-379, 2019.

Abstract

Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect) classification. In particular, we focus on eliciting binary classification performance metrics from pairwise feedback, where a practitioner is queried to provide relative preference between two classifiers. By exploiting key geometric properties of the space of confusion matrices, we obtain provably query efficient algorithms for eliciting linear and linear-fractional performance metrics. We further show that our method is robust to feedback and finite sample noise.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-hiranandani19a, title = {Performance Metric Elicitation from Pairwise Classifier Comparisons}, author = {Hiranandani, Gaurush and Boodaghians, Shant and Mehta, Ruta and Koyejo, Oluwasanmi}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {371--379}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/hiranandani19a/hiranandani19a.pdf}, url = {https://proceedings.mlr.press/v89/hiranandani19a.html}, abstract = {Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect) classification. In particular, we focus on eliciting binary classification performance metrics from pairwise feedback, where a practitioner is queried to provide relative preference between two classifiers. By exploiting key geometric properties of the space of confusion matrices, we obtain provably query efficient algorithms for eliciting linear and linear-fractional performance metrics. We further show that our method is robust to feedback and finite sample noise.} }
Endnote
%0 Conference Paper %T Performance Metric Elicitation from Pairwise Classifier Comparisons %A Gaurush Hiranandani %A Shant Boodaghians %A Ruta Mehta %A Oluwasanmi Koyejo %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-hiranandani19a %I PMLR %P 371--379 %U https://proceedings.mlr.press/v89/hiranandani19a.html %V 89 %X Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect) classification. In particular, we focus on eliciting binary classification performance metrics from pairwise feedback, where a practitioner is queried to provide relative preference between two classifiers. By exploiting key geometric properties of the space of confusion matrices, we obtain provably query efficient algorithms for eliciting linear and linear-fractional performance metrics. We further show that our method is robust to feedback and finite sample noise.
APA
Hiranandani, G., Boodaghians, S., Mehta, R. & Koyejo, O.. (2019). Performance Metric Elicitation from Pairwise Classifier Comparisons. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:371-379 Available from https://proceedings.mlr.press/v89/hiranandani19a.html.

Related Material