No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D. Bagdanov, Joost Van De Weijer
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:43177-43199, 2025.

Abstract

Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices – weight update matrices applied to a pre-trained model – that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-marczak25a, title = {No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces}, author = {Marczak, Daniel and Magistri, Simone and Cygert, Sebastian and Twardowski, Bart{\l}omiej and Bagdanov, Andrew D. and Van De Weijer, Joost}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {43177--43199}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/marczak25a/marczak25a.pdf}, url = {https://proceedings.mlr.press/v267/marczak25a.html}, abstract = {Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices – weight update matrices applied to a pre-trained model – that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training.} }
Endnote
%0 Conference Paper %T No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces %A Daniel Marczak %A Simone Magistri %A Sebastian Cygert %A Bartłomiej Twardowski %A Andrew D. Bagdanov %A Joost Van De Weijer %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-marczak25a %I PMLR %P 43177--43199 %U https://proceedings.mlr.press/v267/marczak25a.html %V 267 %X Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices – weight update matrices applied to a pre-trained model – that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training.
APA
Marczak, D., Magistri, S., Cygert, S., Twardowski, B., Bagdanov, A.D. & Van De Weijer, J.. (2025). No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:43177-43199 Available from https://proceedings.mlr.press/v267/marczak25a.html.

Related Material