Detecting and Correcting for Label Shift with Black Box Predictors

Zachary Lipton, Yu-Xiang Wang, Alexander Smola
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:3122-3130, 2018.

Abstract

Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets), cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the conditional p(x| y) does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution p(y). BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE’s consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-lipton18a, title = {Detecting and Correcting for Label Shift with Black Box Predictors}, author = {Lipton, Zachary and Wang, Yu-Xiang and Smola, Alexander}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {3122--3130}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/lipton18a/lipton18a.pdf}, url = {https://proceedings.mlr.press/v80/lipton18a.html}, abstract = {Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets), cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the conditional p(x| y) does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution p(y). BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE’s consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.} }
Endnote
%0 Conference Paper %T Detecting and Correcting for Label Shift with Black Box Predictors %A Zachary Lipton %A Yu-Xiang Wang %A Alexander Smola %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-lipton18a %I PMLR %P 3122--3130 %U https://proceedings.mlr.press/v80/lipton18a.html %V 80 %X Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets), cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the conditional p(x| y) does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution p(y). BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE’s consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.
APA
Lipton, Z., Wang, Y. & Smola, A.. (2018). Detecting and Correcting for Label Shift with Black Box Predictors. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:3122-3130 Available from https://proceedings.mlr.press/v80/lipton18a.html.

Related Material