Efficient Distributed MLLM Training with Cornstarch

Insu Jang, Runyu Lu, Nikhil Bansal, Ang Chen, Mosharaf Chowdhury
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:50732-50746, 2026.

Abstract

Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training, especially in distributed training. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM’s unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by $2.26\times$ on average in terms of MLLM training throughput. Cornstarch is an open-source project and available on Github.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-jang26g, title = {Efficient Distributed {MLLM} Training with Cornstarch}, author = {Jang, Insu and Lu, Runyu and Bansal, Nikhil and Chen, Ang and Chowdhury, Mosharaf}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {50732--50746}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/jang26g/jang26g.pdf}, url = {https://proceedings.mlr.press/v306/jang26g.html}, abstract = {Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training, especially in distributed training. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM’s unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by $2.26\times$ on average in terms of MLLM training throughput. Cornstarch is an open-source project and available on Github.} }
Endnote
%0 Conference Paper %T Efficient Distributed MLLM Training with Cornstarch %A Insu Jang %A Runyu Lu %A Nikhil Bansal %A Ang Chen %A Mosharaf Chowdhury %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-jang26g %I PMLR %P 50732--50746 %U https://proceedings.mlr.press/v306/jang26g.html %V 306 %X Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training, especially in distributed training. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM’s unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by $2.26\times$ on average in terms of MLLM training throughput. Cornstarch is an open-source project and available on Github.
APA
Jang, I., Lu, R., Bansal, N., Chen, A. & Chowdhury, M.. (2026). Efficient Distributed MLLM Training with Cornstarch. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:50732-50746 Available from https://proceedings.mlr.press/v306/jang26g.html.

Related Material