AutoSampling: Search for Effective Data Sampling Schedules

Ming Sun, Haoxuan Dou, Baopu Li, Junjie Yan, Wanli Ouyang, Lei Cui
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9923-9933, 2021.

Abstract

Data sampling acts as a pivotal role in training deep learning models. However, an effective sampling schedule is difficult to learn due to its inherent high-dimension as a hyper-parameter. In this paper, we propose an AutoSampling method to automatically learn sampling schedules for model training, which consists of the multi-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-sun21a, title = {AutoSampling: Search for Effective Data Sampling Schedules}, author = {Sun, Ming and Dou, Haoxuan and Li, Baopu and Yan, Junjie and Ouyang, Wanli and Cui, Lei}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {9923--9933}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/sun21a/sun21a.pdf}, url = {https://proceedings.mlr.press/v139/sun21a.html}, abstract = {Data sampling acts as a pivotal role in training deep learning models. However, an effective sampling schedule is difficult to learn due to its inherent high-dimension as a hyper-parameter. In this paper, we propose an AutoSampling method to automatically learn sampling schedules for model training, which consists of the multi-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.} }
Endnote
%0 Conference Paper %T AutoSampling: Search for Effective Data Sampling Schedules %A Ming Sun %A Haoxuan Dou %A Baopu Li %A Junjie Yan %A Wanli Ouyang %A Lei Cui %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-sun21a %I PMLR %P 9923--9933 %U https://proceedings.mlr.press/v139/sun21a.html %V 139 %X Data sampling acts as a pivotal role in training deep learning models. However, an effective sampling schedule is difficult to learn due to its inherent high-dimension as a hyper-parameter. In this paper, we propose an AutoSampling method to automatically learn sampling schedules for model training, which consists of the multi-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.
APA
Sun, M., Dou, H., Li, B., Yan, J., Ouyang, W. & Cui, L.. (2021). AutoSampling: Search for Effective Data Sampling Schedules. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:9923-9933 Available from https://proceedings.mlr.press/v139/sun21a.html.

Related Material