Label-Descriptive Patterns and Their Application to Characterizing Classification Errors

Michael A. Hedderich, Jonas Fischer, Dietrich Klakow, Jilles Vreeken
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:8691-8707, 2022.

Abstract

State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors, but also gives a way to act and improve the classifier. We propose to discover those feature-value combinations (i.e., patterns) that strongly correlate with correct resp. erroneous predictions to obtain a global and interpretable description for arbitrary classifiers. We show this is an instance of the more general label description problem, which we formulate in terms of the Minimum Description Length principle. To discover a good pattern set, we develop the efficient Premise algorithm. Through an extensive set of experiments we show it performs very well in practice on both synthetic and real-world data. Unlike existing solutions, it ably recovers ground truth patterns, even on highly imbalanced data over many features. Through two case studies on Visual Question Answering and Named Entity Recognition, we confirm that Premise gives clear and actionable insight into the systematic errors made by modern NLP classifiers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-hedderich22a, title = {Label-Descriptive Patterns and Their Application to Characterizing Classification Errors}, author = {Hedderich, Michael A. and Fischer, Jonas and Klakow, Dietrich and Vreeken, Jilles}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {8691--8707}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/hedderich22a/hedderich22a.pdf}, url = {https://proceedings.mlr.press/v162/hedderich22a.html}, abstract = {State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors, but also gives a way to act and improve the classifier. We propose to discover those feature-value combinations (i.e., patterns) that strongly correlate with correct resp. erroneous predictions to obtain a global and interpretable description for arbitrary classifiers. We show this is an instance of the more general label description problem, which we formulate in terms of the Minimum Description Length principle. To discover a good pattern set, we develop the efficient Premise algorithm. Through an extensive set of experiments we show it performs very well in practice on both synthetic and real-world data. Unlike existing solutions, it ably recovers ground truth patterns, even on highly imbalanced data over many features. Through two case studies on Visual Question Answering and Named Entity Recognition, we confirm that Premise gives clear and actionable insight into the systematic errors made by modern NLP classifiers.} }
Endnote
%0 Conference Paper %T Label-Descriptive Patterns and Their Application to Characterizing Classification Errors %A Michael A. Hedderich %A Jonas Fischer %A Dietrich Klakow %A Jilles Vreeken %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-hedderich22a %I PMLR %P 8691--8707 %U https://proceedings.mlr.press/v162/hedderich22a.html %V 162 %X State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors, but also gives a way to act and improve the classifier. We propose to discover those feature-value combinations (i.e., patterns) that strongly correlate with correct resp. erroneous predictions to obtain a global and interpretable description for arbitrary classifiers. We show this is an instance of the more general label description problem, which we formulate in terms of the Minimum Description Length principle. To discover a good pattern set, we develop the efficient Premise algorithm. Through an extensive set of experiments we show it performs very well in practice on both synthetic and real-world data. Unlike existing solutions, it ably recovers ground truth patterns, even on highly imbalanced data over many features. Through two case studies on Visual Question Answering and Named Entity Recognition, we confirm that Premise gives clear and actionable insight into the systematic errors made by modern NLP classifiers.
APA
Hedderich, M.A., Fischer, J., Klakow, D. & Vreeken, J.. (2022). Label-Descriptive Patterns and Their Application to Characterizing Classification Errors. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:8691-8707 Available from https://proceedings.mlr.press/v162/hedderich22a.html.

Related Material