Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

Wanlong Fang, Tianle Zhang, Wen Tao, Alvin Chan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:29519-29547, 2026.

Abstract

Understanding how multimodal large language models use different modalities is important for reliable reasoning. We employ Partial Information Decomposition (PID) as a decision-level lens and introduce Sensory PID, a conditional formulation that conditions on language and separates unique, redundant, and synergistic contributions from video and audio. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck: even on audio–visual fusion tasks, decisions remain dominated by modality-unique information, with stronger reliance on vision. Modality-shuffling interventions support this asymmetry, while layer-wise analysis reveals a visual-first computation pattern and instruction perturbations show that late-stage sensory fusion is conditioned by language. Beyond diagnosis, PID-guided sample reweighting provides initial evidence that local diagnostic signals can improve multimodal reasoning and grounding performance. As reference validation, our vision–language analysis broadly corroborates previously reported decision-level PID patterns across tasks, models, interventions, and layers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-fang26x, title = {Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition}, author = {Fang, Wanlong and Zhang, Tianle and Tao, Wen and Chan, Alvin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {29519--29547}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/fang26x/fang26x.pdf}, url = {https://proceedings.mlr.press/v306/fang26x.html}, abstract = {Understanding how multimodal large language models use different modalities is important for reliable reasoning. We employ Partial Information Decomposition (PID) as a decision-level lens and introduce Sensory PID, a conditional formulation that conditions on language and separates unique, redundant, and synergistic contributions from video and audio. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck: even on audio–visual fusion tasks, decisions remain dominated by modality-unique information, with stronger reliance on vision. Modality-shuffling interventions support this asymmetry, while layer-wise analysis reveals a visual-first computation pattern and instruction perturbations show that late-stage sensory fusion is conditioned by language. Beyond diagnosis, PID-guided sample reweighting provides initial evidence that local diagnostic signals can improve multimodal reasoning and grounding performance. As reference validation, our vision–language analysis broadly corroborates previously reported decision-level PID patterns across tasks, models, interventions, and layers.} }
Endnote
%0 Conference Paper %T Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition %A Wanlong Fang %A Tianle Zhang %A Wen Tao %A Alvin Chan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-fang26x %I PMLR %P 29519--29547 %U https://proceedings.mlr.press/v306/fang26x.html %V 306 %X Understanding how multimodal large language models use different modalities is important for reliable reasoning. We employ Partial Information Decomposition (PID) as a decision-level lens and introduce Sensory PID, a conditional formulation that conditions on language and separates unique, redundant, and synergistic contributions from video and audio. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck: even on audio–visual fusion tasks, decisions remain dominated by modality-unique information, with stronger reliance on vision. Modality-shuffling interventions support this asymmetry, while layer-wise analysis reveals a visual-first computation pattern and instruction perturbations show that late-stage sensory fusion is conditioned by language. Beyond diagnosis, PID-guided sample reweighting provides initial evidence that local diagnostic signals can improve multimodal reasoning and grounding performance. As reference validation, our vision–language analysis broadly corroborates previously reported decision-level PID patterns across tasks, models, interventions, and layers.
APA
Fang, W., Zhang, T., Tao, W. & Chan, A.. (2026). Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:29519-29547 Available from https://proceedings.mlr.press/v306/fang26x.html.

Related Material