Predicting Out-of-Distribution Error with the Projection Norm

Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, Jacob Steinhardt
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25721-25746, 2022.

Abstract

We propose a metric—Projection Norm—to predict a model’s performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model’s parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, our approach outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples. Our code is available at \url{https://github.com/yaodongyu/ProjNorm}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-yu22i, title = {Predicting Out-of-Distribution Error with the Projection Norm}, author = {Yu, Yaodong and Yang, Zitong and Wei, Alexander and Ma, Yi and Steinhardt, Jacob}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {25721--25746}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/yu22i/yu22i.pdf}, url = {https://proceedings.mlr.press/v162/yu22i.html}, abstract = {We propose a metric—Projection Norm—to predict a model’s performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model’s parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, our approach outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples. Our code is available at \url{https://github.com/yaodongyu/ProjNorm}.} }
Endnote
%0 Conference Paper %T Predicting Out-of-Distribution Error with the Projection Norm %A Yaodong Yu %A Zitong Yang %A Alexander Wei %A Yi Ma %A Jacob Steinhardt %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-yu22i %I PMLR %P 25721--25746 %U https://proceedings.mlr.press/v162/yu22i.html %V 162 %X We propose a metric—Projection Norm—to predict a model’s performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model’s parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, our approach outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples. Our code is available at \url{https://github.com/yaodongyu/ProjNorm}.
APA
Yu, Y., Yang, Z., Wei, A., Ma, Y. & Steinhardt, J.. (2022). Predicting Out-of-Distribution Error with the Projection Norm. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:25721-25746 Available from https://proceedings.mlr.press/v162/yu22i.html.

Related Material