VisDA-2021 Competition: Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data

Dina Bashkirova, Dan Hendrycks, Donghyun Kim, Haojin Liao, Samarth Mishra, Chandramouli Rajagopalan, Kate Saenko, Kuniaki Saito, Burhan Ul Tayyab, Piotr Teterwak, Ben Usman
Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, PMLR 176:66-79, 2022.

Abstract

Progress in machine learning is typically measured by training and testing a model on samples drawn from the same distribution, i.e. the same domain. This over-estimates future accuracy on out-of-distribution data. The Visual Domain Adaptation (VisDA) 2021 competition tests models’ ability to adapt to novel test distributions and handle distributional shift. We set up unsupervised domain adaptation challenges for image classifiers and evaluate adaptation to novel viewpoints, backgrounds, styles and degradation in quality. Our challenge draws on large-scale publicly available datasets but constructs the evaluation across domains, rather than the traditional in-domain benchmarking. Furthermore, we focus on the difficult “universal" setting where, in addition to input distribution drift, methods encounter missing and/or novel classes in the test set. In this paper, we describe the datasets and evaluation metrics and highlight similarities across top-performing methods that might point to promising future directions in universal domain adaptation research. We hope that the competition will encourage further improvement in machine learning methods’ ability to handle realistic data in many deployment scenarios. http://ai.bu.edu/visda-2021/.

Cite this Paper

BibTeX
@InProceedings{pmlr-v176-bashkirova22a, title = {VisDA-2021 Competition: Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data}, author = {Bashkirova, Dina and Hendrycks, Dan and Kim, Donghyun and Liao, Haojin and Mishra, Samarth and Rajagopalan, Chandramouli and Saenko, Kate and Saito, Kuniaki and Tayyab, Burhan Ul and Teterwak, Piotr and Usman, Ben}, booktitle = {Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track}, pages = {66--79}, year = {2022}, editor = {Kiela, Douwe and Ciccone, Marco and Caputo, Barbara}, volume = {176}, series = {Proceedings of Machine Learning Research}, month = {06--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v176/bashkirova22a/bashkirova22a.pdf}, url = {https://proceedings.mlr.press/v176/bashkirova22a.html}, abstract = {Progress in machine learning is typically measured by training and testing a model on samples drawn from the same distribution, i.e. the same domain. This over-estimates future accuracy on out-of-distribution data. The Visual Domain Adaptation (VisDA) 2021 competition tests models’ ability to adapt to novel test distributions and handle distributional shift. We set up unsupervised domain adaptation challenges for image classifiers and evaluate adaptation to novel viewpoints, backgrounds, styles and degradation in quality. Our challenge draws on large-scale publicly available datasets but constructs the evaluation across domains, rather than the traditional in-domain benchmarking. Furthermore, we focus on the difficult “universal" setting where, in addition to input distribution drift, methods encounter missing and/or novel classes in the test set. In this paper, we describe the datasets and evaluation metrics and highlight similarities across top-performing methods that might point to promising future directions in universal domain adaptation research. We hope that the competition will encourage further improvement in machine learning methods’ ability to handle realistic data in many deployment scenarios. http://ai.bu.edu/visda-2021/.} }
Endnote
%0 Conference Paper %T VisDA-2021 Competition: Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data %A Dina Bashkirova %A Dan Hendrycks %A Donghyun Kim %A Haojin Liao %A Samarth Mishra %A Chandramouli Rajagopalan %A Kate Saenko %A Kuniaki Saito %A Burhan Ul Tayyab %A Piotr Teterwak %A Ben Usman %B Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track %C Proceedings of Machine Learning Research %D 2022 %E Douwe Kiela %E Marco Ciccone %E Barbara Caputo %F pmlr-v176-bashkirova22a %I PMLR %P 66--79 %U https://proceedings.mlr.press/v176/bashkirova22a.html %V 176 %X Progress in machine learning is typically measured by training and testing a model on samples drawn from the same distribution, i.e. the same domain. This over-estimates future accuracy on out-of-distribution data. The Visual Domain Adaptation (VisDA) 2021 competition tests models’ ability to adapt to novel test distributions and handle distributional shift. We set up unsupervised domain adaptation challenges for image classifiers and evaluate adaptation to novel viewpoints, backgrounds, styles and degradation in quality. Our challenge draws on large-scale publicly available datasets but constructs the evaluation across domains, rather than the traditional in-domain benchmarking. Furthermore, we focus on the difficult “universal" setting where, in addition to input distribution drift, methods encounter missing and/or novel classes in the test set. In this paper, we describe the datasets and evaluation metrics and highlight similarities across top-performing methods that might point to promising future directions in universal domain adaptation research. We hope that the competition will encourage further improvement in machine learning methods’ ability to handle realistic data in many deployment scenarios. http://ai.bu.edu/visda-2021/.
APA
Bashkirova, D., Hendrycks, D., Kim, D., Liao, H., Mishra, S., Rajagopalan, C., Saenko, K., Saito, K., Tayyab, B.U., Teterwak, P. & Usman, B.. (2022). VisDA-2021 Competition: Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data. Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, in Proceedings of Machine Learning Research 176:66-79 Available from https://proceedings.mlr.press/v176/bashkirova22a.html.

Related Material