Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification

Junfeng Wen, Chun-Nam Yu, Russell Greiner
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):631-639, 2014.

Abstract

Many learning situations involve learning the conditional distribution p(y|x) when the training instances are drawn from the training distribution p_tr(x), even though it will later be used to predict for instances drawn from a different test distribution p_te(x). Most current approaches focus on learning how to reweigh the training examples, to make them resemble the test distribution. However, reweighing does not always help, because (we show that) the test error also depends on the correctness of the underlying model class. This paper analyses this situation by viewing the problem of learning under changing distributions as a game between a learner and an adversary. We characterize when such reweighing is needed, and also provide an algorithm, robust covariate shift adjustment (RCSA), that provides relevant weights. Our empirical studies, on UCI datasets and a real-world cancer prognostic prediction dataset, show that our analysis applies, and that our RCSA works effectively.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-wen14, title = {Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification}, author = {Wen, Junfeng and Yu, Chun-Nam and Greiner, Russell}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {631--639}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/wen14.pdf}, url = {https://proceedings.mlr.press/v32/wen14.html}, abstract = {Many learning situations involve learning the conditional distribution p(y|x) when the training instances are drawn from the training distribution p_tr(x), even though it will later be used to predict for instances drawn from a different test distribution p_te(x). Most current approaches focus on learning how to reweigh the training examples, to make them resemble the test distribution. However, reweighing does not always help, because (we show that) the test error also depends on the correctness of the underlying model class. This paper analyses this situation by viewing the problem of learning under changing distributions as a game between a learner and an adversary. We characterize when such reweighing is needed, and also provide an algorithm, robust covariate shift adjustment (RCSA), that provides relevant weights. Our empirical studies, on UCI datasets and a real-world cancer prognostic prediction dataset, show that our analysis applies, and that our RCSA works effectively.} }
Endnote
%0 Conference Paper %T Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification %A Junfeng Wen %A Chun-Nam Yu %A Russell Greiner %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-wen14 %I PMLR %P 631--639 %U https://proceedings.mlr.press/v32/wen14.html %V 32 %N 2 %X Many learning situations involve learning the conditional distribution p(y|x) when the training instances are drawn from the training distribution p_tr(x), even though it will later be used to predict for instances drawn from a different test distribution p_te(x). Most current approaches focus on learning how to reweigh the training examples, to make them resemble the test distribution. However, reweighing does not always help, because (we show that) the test error also depends on the correctness of the underlying model class. This paper analyses this situation by viewing the problem of learning under changing distributions as a game between a learner and an adversary. We characterize when such reweighing is needed, and also provide an algorithm, robust covariate shift adjustment (RCSA), that provides relevant weights. Our empirical studies, on UCI datasets and a real-world cancer prognostic prediction dataset, show that our analysis applies, and that our RCSA works effectively.
RIS
TY - CPAPER TI - Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification AU - Junfeng Wen AU - Chun-Nam Yu AU - Russell Greiner BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-wen14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 631 EP - 639 L1 - http://proceedings.mlr.press/v32/wen14.pdf UR - https://proceedings.mlr.press/v32/wen14.html AB - Many learning situations involve learning the conditional distribution p(y|x) when the training instances are drawn from the training distribution p_tr(x), even though it will later be used to predict for instances drawn from a different test distribution p_te(x). Most current approaches focus on learning how to reweigh the training examples, to make them resemble the test distribution. However, reweighing does not always help, because (we show that) the test error also depends on the correctness of the underlying model class. This paper analyses this situation by viewing the problem of learning under changing distributions as a game between a learner and an adversary. We characterize when such reweighing is needed, and also provide an algorithm, robust covariate shift adjustment (RCSA), that provides relevant weights. Our empirical studies, on UCI datasets and a real-world cancer prognostic prediction dataset, show that our analysis applies, and that our RCSA works effectively. ER -
APA
Wen, J., Yu, C. & Greiner, R.. (2014). Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):631-639 Available from https://proceedings.mlr.press/v32/wen14.html.

Related Material