Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space

Long Minh Bui, Tuan Anh Le Van, Tung Phi Duc, Phi Le Nguyen, Jana Doppa, Trong Nghia Hoang
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:819-838, 2026.

Abstract

Model merging aims to combine existing single-task solutions into a multi-task solution without additional data-driven fine-tuning. Most existing approaches achieve this using geometric properties of local solution spaces. However, such geometric views provide limited guidance for scoring how statistically useful each task-specific update direction is across tasks during merging. To address this, we formulate model merging from a new perspective of probabilistic inference under a product-of-experts ({PoE}) scenario where each single-task solution defines an energy-based expert model ({EBM}) over the merged parameters. We show that several existing model merging methods arise as special cases of our framework under energy designs that impose implicit {Gaussian} assumptions on directional residuals between merged and task-specific models. Empirically, we find that these residuals are often heavy-tailed which exposes a mismatch with the imposed light-tailed {Gaussian} structures. We address this with a heavy-tailed {PoE} design based on {Cauchy} experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure. Experiments across multiple tasks and architectures show significant improvements over state-of-the-arts baselines. Our code is available at https://github.com/MinhLong210/{PoE}-{EBM}-Merging.git.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-bui26a, title = {Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space}, author = {Bui, Long Minh and Van, Tuan Anh Le and Duc, Tung Phi and Nguyen, Phi Le and Doppa, Jana and Hoang, Trong Nghia}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {819--838}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/bui26a/bui26a.pdf}, url = {https://proceedings.mlr.press/v337/bui26a.html}, abstract = {Model merging aims to combine existing single-task solutions into a multi-task solution without additional data-driven fine-tuning. Most existing approaches achieve this using geometric properties of local solution spaces. However, such geometric views provide limited guidance for scoring how statistically useful each task-specific update direction is across tasks during merging. To address this, we formulate model merging from a new perspective of probabilistic inference under a product-of-experts ({PoE}) scenario where each single-task solution defines an energy-based expert model ({EBM}) over the merged parameters. We show that several existing model merging methods arise as special cases of our framework under energy designs that impose implicit {Gaussian} assumptions on directional residuals between merged and task-specific models. Empirically, we find that these residuals are often heavy-tailed which exposes a mismatch with the imposed light-tailed {Gaussian} structures. We address this with a heavy-tailed {PoE} design based on {Cauchy} experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure. Experiments across multiple tasks and architectures show significant improvements over state-of-the-arts baselines. Our code is available at https://github.com/MinhLong210/{PoE}-{EBM}-Merging.git.} }
Endnote
%0 Conference Paper %T Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space %A Long Minh Bui %A Tuan Anh Le Van %A Tung Phi Duc %A Phi Le Nguyen %A Jana Doppa %A Trong Nghia Hoang %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-bui26a %I PMLR %P 819--838 %U https://proceedings.mlr.press/v337/bui26a.html %V 337 %X Model merging aims to combine existing single-task solutions into a multi-task solution without additional data-driven fine-tuning. Most existing approaches achieve this using geometric properties of local solution spaces. However, such geometric views provide limited guidance for scoring how statistically useful each task-specific update direction is across tasks during merging. To address this, we formulate model merging from a new perspective of probabilistic inference under a product-of-experts ({PoE}) scenario where each single-task solution defines an energy-based expert model ({EBM}) over the merged parameters. We show that several existing model merging methods arise as special cases of our framework under energy designs that impose implicit {Gaussian} assumptions on directional residuals between merged and task-specific models. Empirically, we find that these residuals are often heavy-tailed which exposes a mismatch with the imposed light-tailed {Gaussian} structures. We address this with a heavy-tailed {PoE} design based on {Cauchy} experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure. Experiments across multiple tasks and architectures show significant improvements over state-of-the-arts baselines. Our code is available at https://github.com/MinhLong210/{PoE}-{EBM}-Merging.git.
APA
Bui, L.M., Van, T.A.L., Duc, T.P., Nguyen, P.L., Doppa, J. & Hoang, T.N.. (2026). Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:819-838 Available from https://proceedings.mlr.press/v337/bui26a.html.

Related Material