Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation

Nick Cooper, Lijun Chen, Sailesh Dwivedy, Danna Gurari
Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, PMLR 282:133-159, 2026.

Abstract

Knowledge distillation (KD) methods transfer the knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student’s backbone using feature-based losses \emph{exclusively} (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a \emph{knowledge quality metric} for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to $15$% over standard approaches. We publicly share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.git .

Cite this Paper

BibTeX
@InProceedings{pmlr-v282-cooper26a, title = {Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation}, author = {Cooper, Nick and Chen, Lijun and Dwivedy, Sailesh and Gurari, Danna}, booktitle = {Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations}, pages = {133--159}, year = {2026}, editor = {Acosta, Francisco and Azeglio, Simone and Tolooshams, Bahareh and van de Geijn, Chase and Shewmake, Christian and Sanborn, Sophia and Miolane, Nina}, volume = {282}, series = {Proceedings of Machine Learning Research}, month = {14 Dec 2024--07 Dec 2025}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v282/main/assets/cooper26a/cooper26a.pdf}, url = {https://proceedings.mlr.press/v282/cooper26a.html}, abstract = {Knowledge distillation (KD) methods transfer the knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student’s backbone using feature-based losses \emph{exclusively} (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a \emph{knowledge quality metric} for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to $15$% over standard approaches. We publicly share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.git .} }
Endnote
%0 Conference Paper %T Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation %A Nick Cooper %A Lijun Chen %A Sailesh Dwivedy %A Danna Gurari %B Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations %C Proceedings of Machine Learning Research %D 2026 %E Francisco Acosta %E Simone Azeglio %E Bahareh Tolooshams %E Chase van de Geijn %E Christian Shewmake %E Sophia Sanborn %E Nina Miolane %F pmlr-v282-cooper26a %I PMLR %P 133--159 %U https://proceedings.mlr.press/v282/cooper26a.html %V 282 %X Knowledge distillation (KD) methods transfer the knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student’s backbone using feature-based losses \emph{exclusively} (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a \emph{knowledge quality metric} for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to $15$% over standard approaches. We publicly share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.git .
APA
Cooper, N., Chen, L., Dwivedy, S. & Gurari, D.. (2026). Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation. Proceedings of the 4th (2025) and 3rd (2024) NeurIPS Workshops on Symmetry and Geometry in Neural Representations, in Proceedings of Machine Learning Research 282:133-159 Available from https://proceedings.mlr.press/v282/cooper26a.html.

Related Material