Online Active Model Selection for Pre-trained Classifiers

Mohammad Reza Karimi, Nezihe Merve Gürel, Bojan Karlaš, Johannes Rausch, Ce Zhang, Andreas Krause
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:307-315, 2021.

Abstract

Given $k$ pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can also be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.

Cite this Paper


BibTeX
@InProceedings{pmlr-v130-reza-karimi21a, title = { Online Active Model Selection for Pre-trained Classifiers }, author = {Reza Karimi, Mohammad and Merve G{\"u}rel, Nezihe and Karla\v{s}, Bojan and Rausch, Johannes and Zhang, Ce and Krause, Andreas}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {307--315}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/reza-karimi21a/reza-karimi21a.pdf}, url = {https://proceedings.mlr.press/v130/reza-karimi21a.html}, abstract = { Given $k$ pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can also be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies. } }
Endnote
%0 Conference Paper %T Online Active Model Selection for Pre-trained Classifiers %A Mohammad Reza Karimi %A Nezihe Merve Gürel %A Bojan Karlaš %A Johannes Rausch %A Ce Zhang %A Andreas Krause %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-reza-karimi21a %I PMLR %P 307--315 %U https://proceedings.mlr.press/v130/reza-karimi21a.html %V 130 %X Given $k$ pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can also be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.
APA
Reza Karimi, M., Merve Gürel, N., Karlaš, B., Rausch, J., Zhang, C. & Krause, A.. (2021). Online Active Model Selection for Pre-trained Classifiers . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:307-315 Available from https://proceedings.mlr.press/v130/reza-karimi21a.html.

Related Material