All models are wrong, some are useful: Model Selection with Limited Labels

Patrik Okanovic, Andreas Kirsch, Jannes Kasper, Torsten Hoefler, Andreas Krause, Nezihe Merve Gürel
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:2035-2043, 2025.

Abstract

We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-okanovic25a, title = {All models are wrong, some are useful: Model Selection with Limited Labels}, author = {Okanovic, Patrik and Kirsch, Andreas and Kasper, Jannes and Hoefler, Torsten and Krause, Andreas and G{\"u}rel, Nezihe Merve}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {2035--2043}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/okanovic25a/okanovic25a.pdf}, url = {https://proceedings.mlr.press/v258/okanovic25a.html}, abstract = {We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.} }
Endnote
%0 Conference Paper %T All models are wrong, some are useful: Model Selection with Limited Labels %A Patrik Okanovic %A Andreas Kirsch %A Jannes Kasper %A Torsten Hoefler %A Andreas Krause %A Nezihe Merve Gürel %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-okanovic25a %I PMLR %P 2035--2043 %U https://proceedings.mlr.press/v258/okanovic25a.html %V 258 %X We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.
APA
Okanovic, P., Kirsch, A., Kasper, J., Hoefler, T., Krause, A. & Gürel, N.M.. (2025). All models are wrong, some are useful: Model Selection with Limited Labels. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:2035-2043 Available from https://proceedings.mlr.press/v258/okanovic25a.html.

Related Material