Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

Minsik Choi, Geewook Kim
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:20068-20099, 2026.

Abstract

Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture—matching or exceeding centralized joint training with minimal cost overhead—and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-choi26g, title = {Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging}, author = {Choi, Minsik and Kim, Geewook}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {20068--20099}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/choi26g/choi26g.pdf}, url = {https://proceedings.mlr.press/v306/choi26g.html}, abstract = {Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture—matching or exceeding centralized joint training with minimal cost overhead—and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.} }
Endnote
%0 Conference Paper %T Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging %A Minsik Choi %A Geewook Kim %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-choi26g %I PMLR %P 20068--20099 %U https://proceedings.mlr.press/v306/choi26g.html %V 306 %X Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. These results directly motivate MERIT, a decentralized merge-ready instruction-tuning pipeline that estimates dataset-level gradient conflicts, partitions the mixture along the top PCA conflict axes, fine-tunes each partition independently with no inter-partition communication, and merges once via token-weighted averaging. On Qwen2.5-VL-3B with 136 Vision-FLAN tasks, MERIT improves the 8-benchmark average from 54.3 (joint training) to 57.0. The same recipe scales to a 7B model on a 1.6M-example, 176-source mixture—matching or exceeding centralized joint training with minimal cost overhead—and transfers to text-only FLAN. Our code is available at https://github.com/naver-ai/merit.
APA
Choi, M. & Kim, G.. (2026). Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:20068-20099 Available from https://proceedings.mlr.press/v306/choi26g.html.

Related Material