Ensemble Distribution Distillation via Flow Matching

Jonggeon Park, Giung Nam, Hyunsu Kim, Jongmin Yoon, Juho Lee
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:48170-48191, 2025.

Abstract

Neural network ensembles have proven effective in improving performance across a range of tasks; however, their high computational cost limits their applicability in resource-constrained environments or for large models. Ensemble distillation, the process of transferring knowledge from an ensemble teacher to a smaller student model, offers a promising solution to this challenge. The key is to ensure that the student model is both cost-efficient and achieves performance comparable to the ensemble teacher. With this in mind, we propose a novel ensemble distribution distillation method, which leverages flow matching to effectively transfer the diversity from the ensemble teacher to the student model. Our extensive experiments demonstrate the effectiveness of our proposed method compared to existing ensemble distillation approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-park25i, title = {Ensemble Distribution Distillation via Flow Matching}, author = {Park, Jonggeon and Nam, Giung and Kim, Hyunsu and Yoon, Jongmin and Lee, Juho}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {48170--48191}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/park25i/park25i.pdf}, url = {https://proceedings.mlr.press/v267/park25i.html}, abstract = {Neural network ensembles have proven effective in improving performance across a range of tasks; however, their high computational cost limits their applicability in resource-constrained environments or for large models. Ensemble distillation, the process of transferring knowledge from an ensemble teacher to a smaller student model, offers a promising solution to this challenge. The key is to ensure that the student model is both cost-efficient and achieves performance comparable to the ensemble teacher. With this in mind, we propose a novel ensemble distribution distillation method, which leverages flow matching to effectively transfer the diversity from the ensemble teacher to the student model. Our extensive experiments demonstrate the effectiveness of our proposed method compared to existing ensemble distillation approaches.} }
Endnote
%0 Conference Paper %T Ensemble Distribution Distillation via Flow Matching %A Jonggeon Park %A Giung Nam %A Hyunsu Kim %A Jongmin Yoon %A Juho Lee %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-park25i %I PMLR %P 48170--48191 %U https://proceedings.mlr.press/v267/park25i.html %V 267 %X Neural network ensembles have proven effective in improving performance across a range of tasks; however, their high computational cost limits their applicability in resource-constrained environments or for large models. Ensemble distillation, the process of transferring knowledge from an ensemble teacher to a smaller student model, offers a promising solution to this challenge. The key is to ensure that the student model is both cost-efficient and achieves performance comparable to the ensemble teacher. With this in mind, we propose a novel ensemble distribution distillation method, which leverages flow matching to effectively transfer the diversity from the ensemble teacher to the student model. Our extensive experiments demonstrate the effectiveness of our proposed method compared to existing ensemble distillation approaches.
APA
Park, J., Nam, G., Kim, H., Yoon, J. & Lee, J.. (2025). Ensemble Distribution Distillation via Flow Matching. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:48170-48191 Available from https://proceedings.mlr.press/v267/park25i.html.

Related Material