Simple and Fast Group Robustness by Automatic Feature Reweighting

Shikai Qiu, Andres Potapczynski, Pavel Izmailov, Andrew Gordon Wilson
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:28448-28467, 2023.

Abstract

A major challenge to out-of-distribution generalization is reliance on spurious features — patterns that are predictive of the class label in the training data distribution, but not causally related to the target. Standard methods for reducing the reliance on spurious features typically assume that we know what the spurious feature is, which is rarely true in the real world. Methods that attempt to alleviate this limitation are complex, hard to tune, and lead to a significant computational overhead compared to standard training. In this paper, we propose Automatic Feature Reweighting (AFR), an extremely simple and fast method for updating the model to reduce the reliance on spurious features. AFR retrains the last layer of a standard ERM-trained base model with a weighted loss that emphasizes the examples where the ERM model predicts poorly, automatically upweighting the minority group without group labels. With this simple procedure, we improve upon the best reported results among competing methods trained without spurious attributes on several vision and natural language classification benchmarks, using only a fraction of their compute.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-qiu23c, title = {Simple and Fast Group Robustness by Automatic Feature Reweighting}, author = {Qiu, Shikai and Potapczynski, Andres and Izmailov, Pavel and Wilson, Andrew Gordon}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {28448--28467}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/qiu23c/qiu23c.pdf}, url = {https://proceedings.mlr.press/v202/qiu23c.html}, abstract = {A major challenge to out-of-distribution generalization is reliance on spurious features — patterns that are predictive of the class label in the training data distribution, but not causally related to the target. Standard methods for reducing the reliance on spurious features typically assume that we know what the spurious feature is, which is rarely true in the real world. Methods that attempt to alleviate this limitation are complex, hard to tune, and lead to a significant computational overhead compared to standard training. In this paper, we propose Automatic Feature Reweighting (AFR), an extremely simple and fast method for updating the model to reduce the reliance on spurious features. AFR retrains the last layer of a standard ERM-trained base model with a weighted loss that emphasizes the examples where the ERM model predicts poorly, automatically upweighting the minority group without group labels. With this simple procedure, we improve upon the best reported results among competing methods trained without spurious attributes on several vision and natural language classification benchmarks, using only a fraction of their compute.} }
Endnote
%0 Conference Paper %T Simple and Fast Group Robustness by Automatic Feature Reweighting %A Shikai Qiu %A Andres Potapczynski %A Pavel Izmailov %A Andrew Gordon Wilson %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-qiu23c %I PMLR %P 28448--28467 %U https://proceedings.mlr.press/v202/qiu23c.html %V 202 %X A major challenge to out-of-distribution generalization is reliance on spurious features — patterns that are predictive of the class label in the training data distribution, but not causally related to the target. Standard methods for reducing the reliance on spurious features typically assume that we know what the spurious feature is, which is rarely true in the real world. Methods that attempt to alleviate this limitation are complex, hard to tune, and lead to a significant computational overhead compared to standard training. In this paper, we propose Automatic Feature Reweighting (AFR), an extremely simple and fast method for updating the model to reduce the reliance on spurious features. AFR retrains the last layer of a standard ERM-trained base model with a weighted loss that emphasizes the examples where the ERM model predicts poorly, automatically upweighting the minority group without group labels. With this simple procedure, we improve upon the best reported results among competing methods trained without spurious attributes on several vision and natural language classification benchmarks, using only a fraction of their compute.
APA
Qiu, S., Potapczynski, A., Izmailov, P. & Wilson, A.G.. (2023). Simple and Fast Group Robustness by Automatic Feature Reweighting. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:28448-28467 Available from https://proceedings.mlr.press/v202/qiu23c.html.

Related Material