Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions

Kartik Ahuja, Karthikeyan Shanmugam, Amit Dhurandhar
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:1270-1278, 2021.

Abstract

Recently, invariant risk minimization (IRM) (Arjovsky et al. 2019) was proposed as a promising solution to address out-of-distribution (OOD) generalization. In Ahuja et al. (2020), it was shown that solving for the Nash equilibria of a new class of “ensemble-games” is equivalent to solving IRM. In this work, we extend the framework in Ahuja et al. (2020) for linear regressions by projecting the ensemble-game on an $\ell_{\infty}$ ball. We show that such projections help achieve non-trivial out-of-distribution guarantees despite not achieving perfect invariance. For linear models with confounders, we prove that Nash equilibria of these games are closer to the ideal OOD solutions than the standard empirical risk minimization (ERM) and we also provide learning algorithms that provably converge to these Nash Equilibria. Empirical comparisons of the proposed approach with the state-of-the-art show consistent gains in achieving OOD solutions in several settings involving anti-causal variables and confounders.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-ahuja21a, title = { Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions }, author = {Ahuja, Kartik and Shanmugam, Karthikeyan and Dhurandhar, Amit}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {1270--1278}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/ahuja21a/ahuja21a.pdf}, url = {https://proceedings.mlr.press/v130/ahuja21a.html}, abstract = { Recently, invariant risk minimization (IRM) (Arjovsky et al. 2019) was proposed as a promising solution to address out-of-distribution (OOD) generalization. In Ahuja et al. (2020), it was shown that solving for the Nash equilibria of a new class of “ensemble-games” is equivalent to solving IRM. In this work, we extend the framework in Ahuja et al. (2020) for linear regressions by projecting the ensemble-game on an $\ell_{\infty}$ ball. We show that such projections help achieve non-trivial out-of-distribution guarantees despite not achieving perfect invariance. For linear models with confounders, we prove that Nash equilibria of these games are closer to the ideal OOD solutions than the standard empirical risk minimization (ERM) and we also provide learning algorithms that provably converge to these Nash Equilibria. Empirical comparisons of the proposed approach with the state-of-the-art show consistent gains in achieving OOD solutions in several settings involving anti-causal variables and confounders. } }
Endnote
%0 Conference Paper %T Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions %A Kartik Ahuja %A Karthikeyan Shanmugam %A Amit Dhurandhar %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-ahuja21a %I PMLR %P 1270--1278 %U https://proceedings.mlr.press/v130/ahuja21a.html %V 130 %X Recently, invariant risk minimization (IRM) (Arjovsky et al. 2019) was proposed as a promising solution to address out-of-distribution (OOD) generalization. In Ahuja et al. (2020), it was shown that solving for the Nash equilibria of a new class of “ensemble-games” is equivalent to solving IRM. In this work, we extend the framework in Ahuja et al. (2020) for linear regressions by projecting the ensemble-game on an $\ell_{\infty}$ ball. We show that such projections help achieve non-trivial out-of-distribution guarantees despite not achieving perfect invariance. For linear models with confounders, we prove that Nash equilibria of these games are closer to the ideal OOD solutions than the standard empirical risk minimization (ERM) and we also provide learning algorithms that provably converge to these Nash Equilibria. Empirical comparisons of the proposed approach with the state-of-the-art show consistent gains in achieving OOD solutions in several settings involving anti-causal variables and confounders.
APA
Ahuja, K., Shanmugam, K. & Dhurandhar, A.. (2021). Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:1270-1278 Available from https://proceedings.mlr.press/v130/ahuja21a.html.

Related Material