Predictive Multiplicity in Classification

Charles Marx, Flavio Calmon, Berk Ustun
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:6765-6774, 2020.

Abstract

Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with conflicting predictions. We introduce measures to evaluate the severity of predictive multiplicity, and develop integer programming tools to compute these measures exactly for linear classification problems. We apply our tools to measure predictive multiplicity in recidivism prediction problems. Our results show that real-world datasets may admit competing models that assign wildly conflicting predictions, and motivate the need to report predictive multiplicity in model development.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-marx20a, title = {Predictive Multiplicity in Classification}, author = {Marx, Charles and Calmon, Flavio and Ustun, Berk}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {6765--6774}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/marx20a/marx20a.pdf}, url = {https://proceedings.mlr.press/v119/marx20a.html}, abstract = {Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with conflicting predictions. We introduce measures to evaluate the severity of predictive multiplicity, and develop integer programming tools to compute these measures exactly for linear classification problems. We apply our tools to measure predictive multiplicity in recidivism prediction problems. Our results show that real-world datasets may admit competing models that assign wildly conflicting predictions, and motivate the need to report predictive multiplicity in model development.} }
Endnote
%0 Conference Paper %T Predictive Multiplicity in Classification %A Charles Marx %A Flavio Calmon %A Berk Ustun %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-marx20a %I PMLR %P 6765--6774 %U https://proceedings.mlr.press/v119/marx20a.html %V 119 %X Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with conflicting predictions. We introduce measures to evaluate the severity of predictive multiplicity, and develop integer programming tools to compute these measures exactly for linear classification problems. We apply our tools to measure predictive multiplicity in recidivism prediction problems. Our results show that real-world datasets may admit competing models that assign wildly conflicting predictions, and motivate the need to report predictive multiplicity in model development.
APA
Marx, C., Calmon, F. & Ustun, B.. (2020). Predictive Multiplicity in Classification. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:6765-6774 Available from https://proceedings.mlr.press/v119/marx20a.html.

Related Material