Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

Jiacheng Cui, Bingkui Tong, Xinyue Bi, Xiaohan Zhao, Jiacheng Liu, Zhiqiang Shen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:21984-22006, 2026.

Abstract

Soft labels from teacher models are a de facto practice for knowledge transfer and large-scale dataset distillation (e.g., SRe$^2$L, LPLD). However, when we limit the number of crops per image to reduce the substantial cost of storing precomputed soft labels, these methods suffer severely from local semantic drift: visually ambiguous crops can cause soft supervision to deviate from the image-level ground-truth semantics, leading to persistent errors and a train–test distribution mismatch. We revisit the overlooked role of hard labels and show that, when properly integrated, they can act as a content-invariant semantic anchor that calibrates such drift. We theoretically analyze the emergence of drift under sparse soft-label supervision and demonstrate that hybridizing hard and soft labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which uses hard labels as intermediate corrective signals while preserving the fine-grained benefits of soft labels. Extensive experiments on dataset distillation and large-scale classification benchmarks show consistent generalization improvements. On ImageNet-1K, our method achieves 42.7% accuracy with only 285M soft-label storage (reduces by 100$\times$), outperforming prior state-of-the-art LPLD by 9.0%. Code is available at https://github.com/Jiacheng8/HALD.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-cui26g, title = {Hard Labels In! {R}ethinking the Role of Hard Labels in Mitigating Local Semantic Drift}, author = {Cui, Jiacheng and Tong, Bingkui and Bi, Xinyue and Zhao, Xiaohan and Liu, Jiacheng and Shen, Zhiqiang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {21984--22006}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cui26g/cui26g.pdf}, url = {https://proceedings.mlr.press/v306/cui26g.html}, abstract = {Soft labels from teacher models are a de facto practice for knowledge transfer and large-scale dataset distillation (e.g., SRe$^2$L, LPLD). However, when we limit the number of crops per image to reduce the substantial cost of storing precomputed soft labels, these methods suffer severely from local semantic drift: visually ambiguous crops can cause soft supervision to deviate from the image-level ground-truth semantics, leading to persistent errors and a train–test distribution mismatch. We revisit the overlooked role of hard labels and show that, when properly integrated, they can act as a content-invariant semantic anchor that calibrates such drift. We theoretically analyze the emergence of drift under sparse soft-label supervision and demonstrate that hybridizing hard and soft labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which uses hard labels as intermediate corrective signals while preserving the fine-grained benefits of soft labels. Extensive experiments on dataset distillation and large-scale classification benchmarks show consistent generalization improvements. On ImageNet-1K, our method achieves 42.7% accuracy with only 285M soft-label storage (reduces by 100$\times$), outperforming prior state-of-the-art LPLD by 9.0%. Code is available at https://github.com/Jiacheng8/HALD.} }
Endnote
%0 Conference Paper %T Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift %A Jiacheng Cui %A Bingkui Tong %A Xinyue Bi %A Xiaohan Zhao %A Jiacheng Liu %A Zhiqiang Shen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cui26g %I PMLR %P 21984--22006 %U https://proceedings.mlr.press/v306/cui26g.html %V 306 %X Soft labels from teacher models are a de facto practice for knowledge transfer and large-scale dataset distillation (e.g., SRe$^2$L, LPLD). However, when we limit the number of crops per image to reduce the substantial cost of storing precomputed soft labels, these methods suffer severely from local semantic drift: visually ambiguous crops can cause soft supervision to deviate from the image-level ground-truth semantics, leading to persistent errors and a train–test distribution mismatch. We revisit the overlooked role of hard labels and show that, when properly integrated, they can act as a content-invariant semantic anchor that calibrates such drift. We theoretically analyze the emergence of drift under sparse soft-label supervision and demonstrate that hybridizing hard and soft labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which uses hard labels as intermediate corrective signals while preserving the fine-grained benefits of soft labels. Extensive experiments on dataset distillation and large-scale classification benchmarks show consistent generalization improvements. On ImageNet-1K, our method achieves 42.7% accuracy with only 285M soft-label storage (reduces by 100$\times$), outperforming prior state-of-the-art LPLD by 9.0%. Code is available at https://github.com/Jiacheng8/HALD.
APA
Cui, J., Tong, B., Bi, X., Zhao, X., Liu, J. & Shen, Z.. (2026). Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:21984-22006 Available from https://proceedings.mlr.press/v306/cui26g.html.

Related Material