Toward Data Efficient Model Merging between Different Datasets without Performance Degradation

Masanori Yamada, Tomoya Yamashita, Shin’ya Yamaguchi, Daiki Chijiwa
Proceedings of the 16th Asian Conference on Machine Learning, PMLR 260:239-254, 2025.

Abstract

Model merging is attracting attention as a novel method for creating a new model by combining the weights of different trained models. While previous studies reported that model merging works well for models trained on a single dataset with different random seeds, model merging between different datasets remains unsolved. In this paper, we attempt to reveal the difficulty in merging such models trained on different datasets and alleviate it. Our empirical analyses show that, in contrast to the single-dataset scenarios, dataset information needs to be accessed to achieve high accuracy when merging models trained on different datasets. However, the requirement to use full datasets not only incurs significant computational costs but also becomes a major limitation when integrating models developed and shared by others. To address this, we demonstrate that dataset reduction techniques, such as coreset selection and dataset condensation, effectively reduce the data requirement for model merging. In our experiments with SPLIT-CIFAR10 model merging, the accuracy is significantly improved by 31% when using the full dataset and 24% when using the sampled subset compared with not using the dataset. Our code is available at https://github.com/MasanoriYamada/re-basin-merge-pytorch.

Cite this Paper

BibTeX
@InProceedings{pmlr-v260-yamada25a, title = {Toward Data Efficient Model Merging between Different Datasets without Performance Degradation}, author = {Yamada, Masanori and Yamashita, Tomoya and Yamaguchi, Shin'ya and Chijiwa, Daiki}, booktitle = {Proceedings of the 16th Asian Conference on Machine Learning}, pages = {239--254}, year = {2025}, editor = {Nguyen, Vu and Lin, Hsuan-Tien}, volume = {260}, series = {Proceedings of Machine Learning Research}, month = {05--08 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v260/main/assets/yamada25a/yamada25a.pdf}, url = {https://proceedings.mlr.press/v260/yamada25a.html}, abstract = {Model merging is attracting attention as a novel method for creating a new model by combining the weights of different trained models. While previous studies reported that model merging works well for models trained on a single dataset with different random seeds, model merging between different datasets remains unsolved. In this paper, we attempt to reveal the difficulty in merging such models trained on different datasets and alleviate it. Our empirical analyses show that, in contrast to the single-dataset scenarios, dataset information needs to be accessed to achieve high accuracy when merging models trained on different datasets. However, the requirement to use full datasets not only incurs significant computational costs but also becomes a major limitation when integrating models developed and shared by others. To address this, we demonstrate that dataset reduction techniques, such as coreset selection and dataset condensation, effectively reduce the data requirement for model merging. In our experiments with SPLIT-CIFAR10 model merging, the accuracy is significantly improved by $31$% when using the full dataset and $24$% when using the sampled subset compared with not using the dataset. Our code is available at https://github.com/MasanoriYamada/re-basin-merge-pytorch.} }
Endnote
%0 Conference Paper %T Toward Data Efficient Model Merging between Different Datasets without Performance Degradation %A Masanori Yamada %A Tomoya Yamashita %A Shin’ya Yamaguchi %A Daiki Chijiwa %B Proceedings of the 16th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Vu Nguyen %E Hsuan-Tien Lin %F pmlr-v260-yamada25a %I PMLR %P 239--254 %U https://proceedings.mlr.press/v260/yamada25a.html %V 260 %X Model merging is attracting attention as a novel method for creating a new model by combining the weights of different trained models. While previous studies reported that model merging works well for models trained on a single dataset with different random seeds, model merging between different datasets remains unsolved. In this paper, we attempt to reveal the difficulty in merging such models trained on different datasets and alleviate it. Our empirical analyses show that, in contrast to the single-dataset scenarios, dataset information needs to be accessed to achieve high accuracy when merging models trained on different datasets. However, the requirement to use full datasets not only incurs significant computational costs but also becomes a major limitation when integrating models developed and shared by others. To address this, we demonstrate that dataset reduction techniques, such as coreset selection and dataset condensation, effectively reduce the data requirement for model merging. In our experiments with SPLIT-CIFAR10 model merging, the accuracy is significantly improved by $31$% when using the full dataset and $24$% when using the sampled subset compared with not using the dataset. Our code is available at https://github.com/MasanoriYamada/re-basin-merge-pytorch.
APA
Yamada, M., Yamashita, T., Yamaguchi, S. & Chijiwa, D.. (2025). Toward Data Efficient Model Merging between Different Datasets without Performance Degradation. Proceedings of the 16th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 260:239-254 Available from https://proceedings.mlr.press/v260/yamada25a.html.

Related Material