A constrained Bayesian approach to out-of-distribution prediction

Ziyu Wang, Binjie Yuan, Jiaxun Lu, Bowen Ding, Yunfeng Shao, Qibin Wu, Jun Zhu
Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, PMLR 216:2248-2258, 2023.

Abstract

Consider the problem of out-of-distribution prediction given data from multiple environments. While a sufficiently diverse collection of training environments will facilitate the identification of an invariant predictor, with an optimal generalization performance, many applications only provide us with a limited number of environments. It is thus necessary to consider adapting to distribution shift using a handful of labeled test samples. We propose a constrained Bayesian approach for this task, which restricts to models with a worst-group training loss above a prespecified threshold. Our method avoids a pathology of the standard Bayesian posterior, which occurs when spurious correlations improve in-distribution prediction. We also show that on certain high-dimensional linear problems, constrained modeling improves the sample efficiency of adaptation. Synthetic and real-world experiments demonstrate the robust performance of our approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v216-wang23f, title = {A constrained {B}ayesian approach to out-of-distribution prediction}, author = {Wang, Ziyu and Yuan, Binjie and Lu, Jiaxun and Ding, Bowen and Shao, Yunfeng and Wu, Qibin and Zhu, Jun}, booktitle = {Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence}, pages = {2248--2258}, year = {2023}, editor = {Evans, Robin J. and Shpitser, Ilya}, volume = {216}, series = {Proceedings of Machine Learning Research}, month = {31 Jul--04 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v216/wang23f/wang23f.pdf}, url = {https://proceedings.mlr.press/v216/wang23f.html}, abstract = {Consider the problem of out-of-distribution prediction given data from multiple environments. While a sufficiently diverse collection of training environments will facilitate the identification of an invariant predictor, with an optimal generalization performance, many applications only provide us with a limited number of environments. It is thus necessary to consider adapting to distribution shift using a handful of labeled test samples. We propose a constrained Bayesian approach for this task, which restricts to models with a worst-group training loss above a prespecified threshold. Our method avoids a pathology of the standard Bayesian posterior, which occurs when spurious correlations improve in-distribution prediction. We also show that on certain high-dimensional linear problems, constrained modeling improves the sample efficiency of adaptation. Synthetic and real-world experiments demonstrate the robust performance of our approach.} }
Endnote
%0 Conference Paper %T A constrained Bayesian approach to out-of-distribution prediction %A Ziyu Wang %A Binjie Yuan %A Jiaxun Lu %A Bowen Ding %A Yunfeng Shao %A Qibin Wu %A Jun Zhu %B Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2023 %E Robin J. Evans %E Ilya Shpitser %F pmlr-v216-wang23f %I PMLR %P 2248--2258 %U https://proceedings.mlr.press/v216/wang23f.html %V 216 %X Consider the problem of out-of-distribution prediction given data from multiple environments. While a sufficiently diverse collection of training environments will facilitate the identification of an invariant predictor, with an optimal generalization performance, many applications only provide us with a limited number of environments. It is thus necessary to consider adapting to distribution shift using a handful of labeled test samples. We propose a constrained Bayesian approach for this task, which restricts to models with a worst-group training loss above a prespecified threshold. Our method avoids a pathology of the standard Bayesian posterior, which occurs when spurious correlations improve in-distribution prediction. We also show that on certain high-dimensional linear problems, constrained modeling improves the sample efficiency of adaptation. Synthetic and real-world experiments demonstrate the robust performance of our approach.
APA
Wang, Z., Yuan, B., Lu, J., Ding, B., Shao, Y., Wu, Q. & Zhu, J.. (2023). A constrained Bayesian approach to out-of-distribution prediction. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 216:2248-2258 Available from https://proceedings.mlr.press/v216/wang23f.html.

Related Material