Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport

Adarsh Subbaswamy, Peter Schulam, Suchi Saria
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:3118-3127, 2019.

Abstract

Classical supervised learning produces unreliable models when training and target distributions differ, with most existing solutions requiring samples from the target domain. We propose a proactive approach which learns a relationship in the training domain that will generalize to the target domain by incorporating prior knowledge of aspects of the data generating process that are expected to differ as expressed in a causal selection diagram. Specifically, we remove variables generated by unstable mechanisms from the joint factorization to yield the Surgery Estimator—an interventional distribution that is invariant to the differences across environments. We prove that the surgery estimator finds stable relationships in strictly more scenarios than previous approaches which only consider conditional relationships, and demonstrate this in simulated experiments. We also evaluate on real world data for which the true causal diagram is unknown, performing competitively against entirely data-driven approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-subbaswamy19a, title = {Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport}, author = {Subbaswamy, Adarsh and Schulam, Peter and Saria, Suchi}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {3118--3127}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/subbaswamy19a/subbaswamy19a.pdf}, url = {https://proceedings.mlr.press/v89/subbaswamy19a.html}, abstract = {Classical supervised learning produces unreliable models when training and target distributions differ, with most existing solutions requiring samples from the target domain. We propose a proactive approach which learns a relationship in the training domain that will generalize to the target domain by incorporating prior knowledge of aspects of the data generating process that are expected to differ as expressed in a causal selection diagram. Specifically, we remove variables generated by unstable mechanisms from the joint factorization to yield the Surgery Estimator—an interventional distribution that is invariant to the differences across environments. We prove that the surgery estimator finds stable relationships in strictly more scenarios than previous approaches which only consider conditional relationships, and demonstrate this in simulated experiments. We also evaluate on real world data for which the true causal diagram is unknown, performing competitively against entirely data-driven approaches.} }
Endnote
%0 Conference Paper %T Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport %A Adarsh Subbaswamy %A Peter Schulam %A Suchi Saria %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-subbaswamy19a %I PMLR %P 3118--3127 %U https://proceedings.mlr.press/v89/subbaswamy19a.html %V 89 %X Classical supervised learning produces unreliable models when training and target distributions differ, with most existing solutions requiring samples from the target domain. We propose a proactive approach which learns a relationship in the training domain that will generalize to the target domain by incorporating prior knowledge of aspects of the data generating process that are expected to differ as expressed in a causal selection diagram. Specifically, we remove variables generated by unstable mechanisms from the joint factorization to yield the Surgery Estimator—an interventional distribution that is invariant to the differences across environments. We prove that the surgery estimator finds stable relationships in strictly more scenarios than previous approaches which only consider conditional relationships, and demonstrate this in simulated experiments. We also evaluate on real world data for which the true causal diagram is unknown, performing competitively against entirely data-driven approaches.
APA
Subbaswamy, A., Schulam, P. & Saria, S.. (2019). Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:3118-3127 Available from https://proceedings.mlr.press/v89/subbaswamy19a.html.

Related Material