Evaluating Self-Supervised Learning via Risk Decomposition

Yann Dubois, Tatsunori Hashimoto, Percy Liang
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:8779-8820, 2023.

Abstract

Self-supervised learning (SSL) is typically evaluated using a single metric (linear probing on ImageNet), which neither provides insight into tradeoffs between models nor highlights how to improve them. To address this, we propose an SSL risk decomposition, which generalizes the classical approximation-estimation decomposition. Our decomposition consists of four error terms: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each term and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main source of errors and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-dubois23a, title = {Evaluating Self-Supervised Learning via Risk Decomposition}, author = {Dubois, Yann and Hashimoto, Tatsunori and Liang, Percy}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {8779--8820}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/dubois23a/dubois23a.pdf}, url = {https://proceedings.mlr.press/v202/dubois23a.html}, abstract = {Self-supervised learning (SSL) is typically evaluated using a single metric (linear probing on ImageNet), which neither provides insight into tradeoffs between models nor highlights how to improve them. To address this, we propose an SSL risk decomposition, which generalizes the classical approximation-estimation decomposition. Our decomposition consists of four error terms: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each term and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main source of errors and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components.} }
Endnote
%0 Conference Paper %T Evaluating Self-Supervised Learning via Risk Decomposition %A Yann Dubois %A Tatsunori Hashimoto %A Percy Liang %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-dubois23a %I PMLR %P 8779--8820 %U https://proceedings.mlr.press/v202/dubois23a.html %V 202 %X Self-supervised learning (SSL) is typically evaluated using a single metric (linear probing on ImageNet), which neither provides insight into tradeoffs between models nor highlights how to improve them. To address this, we propose an SSL risk decomposition, which generalizes the classical approximation-estimation decomposition. Our decomposition consists of four error terms: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each term and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main source of errors and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components.
APA
Dubois, Y., Hashimoto, T. & Liang, P.. (2023). Evaluating Self-Supervised Learning via Risk Decomposition. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:8779-8820 Available from https://proceedings.mlr.press/v202/dubois23a.html.

Related Material