Instance-Level Costs for Nuanced Classifier Evaluation

Kabir Kang, Stephen Mussmann
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:55687-55699, 2026.

Abstract

Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate—models with 5% error rate can achieve 1.8% NEC—revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that the evaluated methods—including cost-aware training, focal loss, and post-hoc thresholding—do not consistently improve NEC across datasets. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs. Code and reproducibility artifacts are available at https://github.com/KabirKang/instance-level-costs-paper-icml.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-kang26m, title = {Instance-Level Costs for Nuanced Classifier Evaluation}, author = {Kang, Kabir and Mussmann, Stephen}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {55687--55699}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/kang26m/kang26m.pdf}, url = {https://proceedings.mlr.press/v306/kang26m.html}, abstract = {Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate—models with 5% error rate can achieve 1.8% NEC—revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that the evaluated methods—including cost-aware training, focal loss, and post-hoc thresholding—do not consistently improve NEC across datasets. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs. Code and reproducibility artifacts are available at https://github.com/KabirKang/instance-level-costs-paper-icml.} }
Endnote
%0 Conference Paper %T Instance-Level Costs for Nuanced Classifier Evaluation %A Kabir Kang %A Stephen Mussmann %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-kang26m %I PMLR %P 55687--55699 %U https://proceedings.mlr.press/v306/kang26m.html %V 306 %X Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate—models with 5% error rate can achieve 1.8% NEC—revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that the evaluated methods—including cost-aware training, focal loss, and post-hoc thresholding—do not consistently improve NEC across datasets. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs. Code and reproducibility artifacts are available at https://github.com/KabirKang/instance-level-costs-paper-icml.
APA
Kang, K. & Mussmann, S.. (2026). Instance-Level Costs for Nuanced Classifier Evaluation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:55687-55699 Available from https://proceedings.mlr.press/v306/kang26m.html.

Related Material