Model Fusion with Kullback-Leibler Divergence

Sebastian Claici, Mikhail Yurochkin, Soumya Ghosh, Justin Solomon
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:2038-2047, 2020.

Abstract

We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-claici20a, title = {Model Fusion with Kullback-Leibler Divergence}, author = {Claici, Sebastian and Yurochkin, Mikhail and Ghosh, Soumya and Solomon, Justin}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {2038--2047}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/claici20a/claici20a.pdf}, url = {https://proceedings.mlr.press/v119/claici20a.html}, abstract = {We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.} }
Endnote
%0 Conference Paper %T Model Fusion with Kullback-Leibler Divergence %A Sebastian Claici %A Mikhail Yurochkin %A Soumya Ghosh %A Justin Solomon %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-claici20a %I PMLR %P 2038--2047 %U https://proceedings.mlr.press/v119/claici20a.html %V 119 %X We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.
APA
Claici, S., Yurochkin, M., Ghosh, S. & Solomon, J.. (2020). Model Fusion with Kullback-Leibler Divergence. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2038-2047 Available from https://proceedings.mlr.press/v119/claici20a.html.

Related Material