[edit]
Instance-Level Costs for Nuanced Classifier Evaluation
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:55687-55699, 2026.
Abstract
Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate—models with 5% error rate can achieve 1.8% NEC—revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that the evaluated methods—including cost-aware training, focal loss, and post-hoc thresholding—do not consistently improve NEC across datasets. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs. Code and reproducibility artifacts are available at https://github.com/KabirKang/instance-level-costs-paper-icml.