Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms

Yilun Zhou, Adithya Renduchintala, Xian Li, Sida Wang, Yashar Mehdad, Asish Ghoshal
Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:1486-1494, 2021.

Abstract

Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. While many algorithms have been proposed, there is little study on what the optimal AL algorithm looks like, which would help researchers understand where their models fall short and iterate on the design. In this paper, we present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks. We present qualitative and quantitative insights into the behaviors of this oracle, comparing and contrasting them with those of various heuristics. Moreover, we are able to consistently improve the heuristics using one particular insight. We hope that our findings can better inform future active learning research. The code is available at https://github.com/YilunZhou/optimal-active-learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v130-zhou21b, title = { Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms }, author = {Zhou, Yilun and Renduchintala, Adithya and Li, Xian and Wang, Sida and Mehdad, Yashar and Ghoshal, Asish}, booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics}, pages = {1486--1494}, year = {2021}, editor = {Banerjee, Arindam and Fukumizu, Kenji}, volume = {130}, series = {Proceedings of Machine Learning Research}, month = {13--15 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v130/zhou21b/zhou21b.pdf}, url = {https://proceedings.mlr.press/v130/zhou21b.html}, abstract = { Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. While many algorithms have been proposed, there is little study on what the optimal AL algorithm looks like, which would help researchers understand where their models fall short and iterate on the design. In this paper, we present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks. We present qualitative and quantitative insights into the behaviors of this oracle, comparing and contrasting them with those of various heuristics. Moreover, we are able to consistently improve the heuristics using one particular insight. We hope that our findings can better inform future active learning research. The code is available at https://github.com/YilunZhou/optimal-active-learning. } }
Endnote
%0 Conference Paper %T Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms %A Yilun Zhou %A Adithya Renduchintala %A Xian Li %A Sida Wang %A Yashar Mehdad %A Asish Ghoshal %B Proceedings of The 24th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2021 %E Arindam Banerjee %E Kenji Fukumizu %F pmlr-v130-zhou21b %I PMLR %P 1486--1494 %U https://proceedings.mlr.press/v130/zhou21b.html %V 130 %X Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. While many algorithms have been proposed, there is little study on what the optimal AL algorithm looks like, which would help researchers understand where their models fall short and iterate on the design. In this paper, we present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks. We present qualitative and quantitative insights into the behaviors of this oracle, comparing and contrasting them with those of various heuristics. Moreover, we are able to consistently improve the heuristics using one particular insight. We hope that our findings can better inform future active learning research. The code is available at https://github.com/YilunZhou/optimal-active-learning.
APA
Zhou, Y., Renduchintala, A., Li, X., Wang, S., Mehdad, Y. & Ghoshal, A.. (2021). Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms . Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 130:1486-1494 Available from https://proceedings.mlr.press/v130/zhou21b.html.

Related Material