Learn to Merge: Meta-Learning for Adaptive Multi-Task Model Merging

Jun Chen, Qin Zhang, Weizhi Zhang, Xiao Luo, Philip S. Yu, Ziyue Qiao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18417-18437, 2026.

Abstract

Model merging in the pretrain-finetune paradigm has proven effective by combining multiple finetuned models into one with multi-task capabilities. However, existing methods rely on fix or manually tuned merging coefficients, making the unified model sensitive to the initial merging strategy and suboptimal for downstream adaptation. Thus, this paper proposed an innovative model merging framework called MetaMerging, a novel meta-learning algorithm to adaptively optimize the merging coefficients to construct a unified model tailored for task-specific adapter training. By simulating adapter updates in an inner loop and meta-optimizing merging coefficients in an outer loop, MetaMerging produces more balanced and generalizable unified models. Extensive experiments on CV and NLP fields show strong performance of MetaMerging on various downstream tasks and demonstrate the effectiveness of meta-learning in our method compared to other parameter merging methods. Our code is available at https://github.com/cjcj46262/MetaMerging.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26gq, title = {Learn to Merge: Meta-Learning for Adaptive Multi-Task Model Merging}, author = {Chen, Jun and Zhang, Qin and Zhang, Weizhi and Luo, Xiao and Yu, Philip S. and Qiao, Ziyue}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18417--18437}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26gq/chen26gq.pdf}, url = {https://proceedings.mlr.press/v306/chen26gq.html}, abstract = {Model merging in the pretrain-finetune paradigm has proven effective by combining multiple finetuned models into one with multi-task capabilities. However, existing methods rely on fix or manually tuned merging coefficients, making the unified model sensitive to the initial merging strategy and suboptimal for downstream adaptation. Thus, this paper proposed an innovative model merging framework called MetaMerging, a novel meta-learning algorithm to adaptively optimize the merging coefficients to construct a unified model tailored for task-specific adapter training. By simulating adapter updates in an inner loop and meta-optimizing merging coefficients in an outer loop, MetaMerging produces more balanced and generalizable unified models. Extensive experiments on CV and NLP fields show strong performance of MetaMerging on various downstream tasks and demonstrate the effectiveness of meta-learning in our method compared to other parameter merging methods. Our code is available at https://github.com/cjcj46262/MetaMerging.} }
Endnote
%0 Conference Paper %T Learn to Merge: Meta-Learning for Adaptive Multi-Task Model Merging %A Jun Chen %A Qin Zhang %A Weizhi Zhang %A Xiao Luo %A Philip S. Yu %A Ziyue Qiao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26gq %I PMLR %P 18417--18437 %U https://proceedings.mlr.press/v306/chen26gq.html %V 306 %X Model merging in the pretrain-finetune paradigm has proven effective by combining multiple finetuned models into one with multi-task capabilities. However, existing methods rely on fix or manually tuned merging coefficients, making the unified model sensitive to the initial merging strategy and suboptimal for downstream adaptation. Thus, this paper proposed an innovative model merging framework called MetaMerging, a novel meta-learning algorithm to adaptively optimize the merging coefficients to construct a unified model tailored for task-specific adapter training. By simulating adapter updates in an inner loop and meta-optimizing merging coefficients in an outer loop, MetaMerging produces more balanced and generalizable unified models. Extensive experiments on CV and NLP fields show strong performance of MetaMerging on various downstream tasks and demonstrate the effectiveness of meta-learning in our method compared to other parameter merging methods. Our code is available at https://github.com/cjcj46262/MetaMerging.
APA
Chen, J., Zhang, Q., Zhang, W., Luo, X., Yu, P.S. & Qiao, Z.. (2026). Learn to Merge: Meta-Learning for Adaptive Multi-Task Model Merging. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18417-18437 Available from https://proceedings.mlr.press/v306/chen26gq.html.

Related Material