Can Subnetwork Structure Be the Key to Out-of-Distribution Generalization?

Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, Aaron Courville
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:12356-12367, 2021.

Abstract

Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and confirm the functional lottery ticket hypothesis: the full network contains a subnetwork with proper structure that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the functional structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-zhang21a, title = {Can Subnetwork Structure Be the Key to Out-of-Distribution Generalization?}, author = {Zhang, Dinghuai and Ahuja, Kartik and Xu, Yilun and Wang, Yisen and Courville, Aaron}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {12356--12367}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/zhang21a/zhang21a.pdf}, url = {https://proceedings.mlr.press/v139/zhang21a.html}, abstract = {Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and confirm the functional lottery ticket hypothesis: the full network contains a subnetwork with proper structure that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the functional structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.} }
Endnote
%0 Conference Paper %T Can Subnetwork Structure Be the Key to Out-of-Distribution Generalization? %A Dinghuai Zhang %A Kartik Ahuja %A Yilun Xu %A Yisen Wang %A Aaron Courville %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-zhang21a %I PMLR %P 12356--12367 %U https://proceedings.mlr.press/v139/zhang21a.html %V 139 %X Can models with particular structure avoid being biased towards spurious correlation in out-of-distribution (OOD) generalization? Peters et al. (2016) provides a positive answer for linear cases. In this paper, we use a functional modular probing method to analyze deep model structures under OOD setting. We demonstrate that even in biased models (which focus on spurious correlation) there still exist unbiased functional subnetworks. Furthermore, we articulate and confirm the functional lottery ticket hypothesis: the full network contains a subnetwork with proper structure that can achieve better OOD performance. We then propose Modular Risk Minimization to solve the subnetwork selection problem. Our algorithm learns the functional structure from a given dataset, and can be combined with any other OOD regularization methods. Experiments on various OOD generalization tasks corroborate the effectiveness of our method.
APA
Zhang, D., Ahuja, K., Xu, Y., Wang, Y. & Courville, A.. (2021). Can Subnetwork Structure Be the Key to Out-of-Distribution Generalization?. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:12356-12367 Available from https://proceedings.mlr.press/v139/zhang21a.html.

Related Material