Diversified Batch Selection for Training Acceleration

Feng Hong, Yueming Lyu, Jiangchao Yao, Ya Zhang, Ivor Tsang, Yanfeng Wang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:18648-18667, 2024.

Abstract

The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-hong24c, title = {Diversified Batch Selection for Training Acceleration}, author = {Hong, Feng and Lyu, Yueming and Yao, Jiangchao and Zhang, Ya and Tsang, Ivor and Wang, Yanfeng}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {18648--18667}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hong24c/hong24c.pdf}, url = {https://proceedings.mlr.press/v235/hong24c.html}, abstract = {The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.} }
Endnote
%0 Conference Paper %T Diversified Batch Selection for Training Acceleration %A Feng Hong %A Yueming Lyu %A Jiangchao Yao %A Ya Zhang %A Ivor Tsang %A Yanfeng Wang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-hong24c %I PMLR %P 18648--18667 %U https://proceedings.mlr.press/v235/hong24c.html %V 235 %X The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.
APA
Hong, F., Lyu, Y., Yao, J., Zhang, Y., Tsang, I. & Wang, Y.. (2024). Diversified Batch Selection for Training Acceleration. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:18648-18667 Available from https://proceedings.mlr.press/v235/hong24c.html.

Related Material