Causal Fine-Tuning under Latent Confounded Shift

Jialin Yu, Yuxiang Zhou, Haoxuan Li, Junchi Yu, Mengyue Yang, Yulan He, Nevin L. Zhang, Philip Torr, Ricardo Silva
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:151776-151797, 2026.

Abstract

Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning (CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-yu26bx, title = {Causal Fine-Tuning under Latent Confounded Shift}, author = {Yu, Jialin and Zhou, Yuxiang and Li, Haoxuan and Yu, Junchi and Yang, Mengyue and He, Yulan and Zhang, Nevin L. and Torr, Philip and Silva, Ricardo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {151776--151797}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yu26bx/yu26bx.pdf}, url = {https://proceedings.mlr.press/v306/yu26bx.html}, abstract = {Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning (CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure.} }
Endnote
%0 Conference Paper %T Causal Fine-Tuning under Latent Confounded Shift %A Jialin Yu %A Yuxiang Zhou %A Haoxuan Li %A Junchi Yu %A Mengyue Yang %A Yulan He %A Nevin L. Zhang %A Philip Torr %A Ricardo Silva %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yu26bx %I PMLR %P 151776--151797 %U https://proceedings.mlr.press/v306/yu26bx.html %V 306 %X Adapting to latent confounded shift remains a core challenge in modern AI. This setting is driven by hidden variables that induce spurious correlations between inputs and outputs during training, leading models to rely on non-causal shortcuts. For example, a model may learn to treat metadata (e.g., data source like "Amazon") as a proxy for positive sentiment, causing failure when the source becomes predominantly negative during deployment. To address this latent confounded shift, we introduce Causal Fine-Tuning (CFT). Using a structural causal model as an inductive bias, we derive sufficient identification conditions that motivate a fine-tuning objective for decomposing representations into high-level stable and low-level shift-sensitive components. Instantiating this framework in BERT, we show that learning such causal/spurious representations and adjusting them accordingly yield a more robust predictor. Experiments on spurious correlation injection attacks in text demonstrate that our method outperforms black-box domain generalization baselines, highlighting the benefits of explicitly modeling causal structure.
APA
Yu, J., Zhou, Y., Li, H., Yu, J., Yang, M., He, Y., Zhang, N.L., Torr, P. & Silva, R.. (2026). Causal Fine-Tuning under Latent Confounded Shift. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:151776-151797 Available from https://proceedings.mlr.press/v306/yu26bx.html.

Related Material