What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?

Weijian Deng, Stephen Gould, Liang Zheng
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2579-2589, 2021.

Abstract

Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson’s Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-deng21a, title = {What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?}, author = {Deng, Weijian and Gould, Stephen and Zheng, Liang}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2579--2589}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/deng21a/deng21a.pdf}, url = {https://proceedings.mlr.press/v139/deng21a.html}, abstract = {Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson’s Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.} }
Endnote
%0 Conference Paper %T What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments? %A Weijian Deng %A Stephen Gould %A Liang Zheng %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-deng21a %I PMLR %P 2579--2589 %U https://proceedings.mlr.press/v139/deng21a.html %V 139 %X Understanding classifier decision under novel environments is central to the community, and a common practice is evaluating it on labeled test sets. However, in real-world testing, image annotations are difficult and expensive to obtain, especially when the test environment is changing. A natural question then arises: given a trained classifier, can we evaluate its accuracy on varying unlabeled test sets? In this work, we train semantic classification and rotation prediction in a multi-task way. On a series of datasets, we report an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson’s Correlation r > 0.88). This finding allows us to utilize linear regression to estimate classifier performance from the accuracy of rotation prediction which can be obtained on the test set through the freely generated rotation labels.
APA
Deng, W., Gould, S. & Zheng, L.. (2021). What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2579-2589 Available from https://proceedings.mlr.press/v139/deng21a.html.

Related Material