Traversing Between Modes in Function Space for Fast Ensembling

Eunggu Yun, Hyungi Lee, Giung Nam, Juho Lee
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:40555-40577, 2023.

Abstract

Deep ensemble is a simple yet powerful way to improve the performance of deep neural networks. Under this motivation, recent works on mode connectivity have shown that parameters of ensembles are connected by low-loss subspaces, and one can efficiently collect ensemble parameters in those subspaces. While this provides a way to efficiently train ensembles, for inference, multiple forward passes should still be executed using all the ensemble parameters, which often becomes a serious bottleneck for real-world deployment. In this work, we propose a novel framework to reduce such costs. Given a low-loss subspace connecting two modes of a neural network, we build an additional neural network that predicts the output of the original neural network evaluated at a certain point in the low-loss subspace. The additional neural network, which we call a “bridge”, is a lightweight network that takes minimal features from the original network and predicts outputs for the low-loss subspace without forward passes through the original network. We empirically demonstrate that we can indeed train such bridge networks and significantly reduce inference costs with the help of bridge networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-yun23a, title = {Traversing Between Modes in Function Space for Fast Ensembling}, author = {Yun, Eunggu and Lee, Hyungi and Nam, Giung and Lee, Juho}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {40555--40577}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/yun23a/yun23a.pdf}, url = {https://proceedings.mlr.press/v202/yun23a.html}, abstract = {Deep ensemble is a simple yet powerful way to improve the performance of deep neural networks. Under this motivation, recent works on mode connectivity have shown that parameters of ensembles are connected by low-loss subspaces, and one can efficiently collect ensemble parameters in those subspaces. While this provides a way to efficiently train ensembles, for inference, multiple forward passes should still be executed using all the ensemble parameters, which often becomes a serious bottleneck for real-world deployment. In this work, we propose a novel framework to reduce such costs. Given a low-loss subspace connecting two modes of a neural network, we build an additional neural network that predicts the output of the original neural network evaluated at a certain point in the low-loss subspace. The additional neural network, which we call a “bridge”, is a lightweight network that takes minimal features from the original network and predicts outputs for the low-loss subspace without forward passes through the original network. We empirically demonstrate that we can indeed train such bridge networks and significantly reduce inference costs with the help of bridge networks.} }
Endnote
%0 Conference Paper %T Traversing Between Modes in Function Space for Fast Ensembling %A Eunggu Yun %A Hyungi Lee %A Giung Nam %A Juho Lee %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-yun23a %I PMLR %P 40555--40577 %U https://proceedings.mlr.press/v202/yun23a.html %V 202 %X Deep ensemble is a simple yet powerful way to improve the performance of deep neural networks. Under this motivation, recent works on mode connectivity have shown that parameters of ensembles are connected by low-loss subspaces, and one can efficiently collect ensemble parameters in those subspaces. While this provides a way to efficiently train ensembles, for inference, multiple forward passes should still be executed using all the ensemble parameters, which often becomes a serious bottleneck for real-world deployment. In this work, we propose a novel framework to reduce such costs. Given a low-loss subspace connecting two modes of a neural network, we build an additional neural network that predicts the output of the original neural network evaluated at a certain point in the low-loss subspace. The additional neural network, which we call a “bridge”, is a lightweight network that takes minimal features from the original network and predicts outputs for the low-loss subspace without forward passes through the original network. We empirically demonstrate that we can indeed train such bridge networks and significantly reduce inference costs with the help of bridge networks.
APA
Yun, E., Lee, H., Nam, G. & Lee, J.. (2023). Traversing Between Modes in Function Space for Fast Ensembling. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:40555-40577 Available from https://proceedings.mlr.press/v202/yun23a.html.

Related Material