PAC Generalization via Invariant Representations

Advait U Parulekar, Karthikeyan Shanmugam, Sanjay Shakkottai
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:27378-27400, 2023.

Abstract

Invariant representations are transformations of the covariates such that the best model on top of the representation is invariant across training environments. In the context of linear Structural Equation Models (SEMs), invariant representations might allow us to learn models with out-of-distribution guarantees, i.e., models that are robust to interventions in the SEM. To address the invariant representation problem in a finite sample setting, we consider the notion of $\epsilon$-approximate invariance. We study the following question: If a representation is approximately invariant with respect to a given number of training interventions, will it continue to be approximately invariant on a larger collection of unseen intervened SEMs? Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees for approximate invariance that holds probabilistically over a family of linear SEMs without faithfulness assumptions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-parulekar23a, title = {{PAC} Generalization via Invariant Representations}, author = {Parulekar, Advait U and Shanmugam, Karthikeyan and Shakkottai, Sanjay}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {27378--27400}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/parulekar23a/parulekar23a.pdf}, url = {https://proceedings.mlr.press/v202/parulekar23a.html}, abstract = {Invariant representations are transformations of the covariates such that the best model on top of the representation is invariant across training environments. In the context of linear Structural Equation Models (SEMs), invariant representations might allow us to learn models with out-of-distribution guarantees, i.e., models that are robust to interventions in the SEM. To address the invariant representation problem in a finite sample setting, we consider the notion of $\epsilon$-approximate invariance. We study the following question: If a representation is approximately invariant with respect to a given number of training interventions, will it continue to be approximately invariant on a larger collection of unseen intervened SEMs? Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees for approximate invariance that holds probabilistically over a family of linear SEMs without faithfulness assumptions.} }
Endnote
%0 Conference Paper %T PAC Generalization via Invariant Representations %A Advait U Parulekar %A Karthikeyan Shanmugam %A Sanjay Shakkottai %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-parulekar23a %I PMLR %P 27378--27400 %U https://proceedings.mlr.press/v202/parulekar23a.html %V 202 %X Invariant representations are transformations of the covariates such that the best model on top of the representation is invariant across training environments. In the context of linear Structural Equation Models (SEMs), invariant representations might allow us to learn models with out-of-distribution guarantees, i.e., models that are robust to interventions in the SEM. To address the invariant representation problem in a finite sample setting, we consider the notion of $\epsilon$-approximate invariance. We study the following question: If a representation is approximately invariant with respect to a given number of training interventions, will it continue to be approximately invariant on a larger collection of unseen intervened SEMs? Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees for approximate invariance that holds probabilistically over a family of linear SEMs without faithfulness assumptions.
APA
Parulekar, A.U., Shanmugam, K. & Shakkottai, S.. (2023). PAC Generalization via Invariant Representations. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:27378-27400 Available from https://proceedings.mlr.press/v202/parulekar23a.html.

Related Material