Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions

Lin Chen, Xiaoke Zhao, Kun Ding, Weiwei Feng, Changtao Miao, Zili Wang, Wenxuan Guo, Ying Wang, Kaiyuan Zheng, Bo Zhang, Zhe Li, Shiming Xiang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18438-18465, 2026.

Abstract

Multimodal Large Language Models (MLLMs) demonstrate impressive cross-modal capabilities, yet their substantial size poses significant deployment challenges. Knowledge distillation (KD) is a promising solution for compressing these models, but existing methods primarily rely on static next-token alignment, neglecting the dynamic token interactions, which embed essential capabilities for multimodal understanding and generation. To this end, we introduce Align-TI, a novel KD framework designed from the perspective of Token Interactions. Our approach is motivated by the insight that MLLMs rely on two primary interactions: vision-instruction token interactions to extract relevant visual information, and intra-response token interactions for coherent generation. Accordingly, Align-TI introduces two components: IVA enables the student model to imitate the teacher’s instruction-relevant visual information extract capability by aligning on salient visual regions. TPA captures the teacher’s dynamic generative logic by aligning the sequential token-to-token transition probabilities. Extensive experiments demonstrate Align-TI’s superiority. Notably, our approach achieves 2.6% relative improvement over Vanilla KD, and our distilled Align-TI-2B even outperforms LLaVA-1.5-7B (a much larger MLLM) by 7.0%, establishing a new state-of-the-art distillation framework for training parameter-efficient MLLMs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26gr, title = {Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions}, author = {Chen, Lin and Zhao, Xiaoke and Ding, Kun and Feng, Weiwei and Miao, Changtao and Wang, Zili and Guo, Wenxuan and Wang, Ying and Zheng, Kaiyuan and Zhang, Bo and Li, Zhe and Xiang, Shiming}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18438--18465}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26gr/chen26gr.pdf}, url = {https://proceedings.mlr.press/v306/chen26gr.html}, abstract = {Multimodal Large Language Models (MLLMs) demonstrate impressive cross-modal capabilities, yet their substantial size poses significant deployment challenges. Knowledge distillation (KD) is a promising solution for compressing these models, but existing methods primarily rely on static next-token alignment, neglecting the dynamic token interactions, which embed essential capabilities for multimodal understanding and generation. To this end, we introduce Align-TI, a novel KD framework designed from the perspective of Token Interactions. Our approach is motivated by the insight that MLLMs rely on two primary interactions: vision-instruction token interactions to extract relevant visual information, and intra-response token interactions for coherent generation. Accordingly, Align-TI introduces two components: IVA enables the student model to imitate the teacher’s instruction-relevant visual information extract capability by aligning on salient visual regions. TPA captures the teacher’s dynamic generative logic by aligning the sequential token-to-token transition probabilities. Extensive experiments demonstrate Align-TI’s superiority. Notably, our approach achieves 2.6% relative improvement over Vanilla KD, and our distilled Align-TI-2B even outperforms LLaVA-1.5-7B (a much larger MLLM) by 7.0%, establishing a new state-of-the-art distillation framework for training parameter-efficient MLLMs.} }
Endnote
%0 Conference Paper %T Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions %A Lin Chen %A Xiaoke Zhao %A Kun Ding %A Weiwei Feng %A Changtao Miao %A Zili Wang %A Wenxuan Guo %A Ying Wang %A Kaiyuan Zheng %A Bo Zhang %A Zhe Li %A Shiming Xiang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26gr %I PMLR %P 18438--18465 %U https://proceedings.mlr.press/v306/chen26gr.html %V 306 %X Multimodal Large Language Models (MLLMs) demonstrate impressive cross-modal capabilities, yet their substantial size poses significant deployment challenges. Knowledge distillation (KD) is a promising solution for compressing these models, but existing methods primarily rely on static next-token alignment, neglecting the dynamic token interactions, which embed essential capabilities for multimodal understanding and generation. To this end, we introduce Align-TI, a novel KD framework designed from the perspective of Token Interactions. Our approach is motivated by the insight that MLLMs rely on two primary interactions: vision-instruction token interactions to extract relevant visual information, and intra-response token interactions for coherent generation. Accordingly, Align-TI introduces two components: IVA enables the student model to imitate the teacher’s instruction-relevant visual information extract capability by aligning on salient visual regions. TPA captures the teacher’s dynamic generative logic by aligning the sequential token-to-token transition probabilities. Extensive experiments demonstrate Align-TI’s superiority. Notably, our approach achieves 2.6% relative improvement over Vanilla KD, and our distilled Align-TI-2B even outperforms LLaVA-1.5-7B (a much larger MLLM) by 7.0%, establishing a new state-of-the-art distillation framework for training parameter-efficient MLLMs.
APA
Chen, L., Zhao, X., Ding, K., Feng, W., Miao, C., Wang, Z., Guo, W., Wang, Y., Zheng, K., Zhang, B., Li, Z. & Xiang, S.. (2026). Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18438-18465 Available from https://proceedings.mlr.press/v306/chen26gr.html.

Related Material