Evaluating and Steering Modality Preferences in Multi-modal LLMs

Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:157593-157623, 2026.

Abstract

Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit modality preference, a tendency to favor one modality over another when processing multi-modal contexts. To study this question, we introduce $MC^2$ benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making. Extensive experiments reveal that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences are statistically associated with performances of downstream taks for MLLMs. Further analysis shows that modality preference can be controlled by instruction guidance and captured within the latent representations of MLLMs. Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning. This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple multi-modal understanding and reasoning tasks.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhang26ed, title = {Evaluating and Steering Modality Preferences in Multi-modal {LLM}s}, author = {Zhang, Yu and Ma, Jinlong and Hou, Yongshuai and Bai, Xuefeng and Chen, Kehai and Xiang, Yang and Yu, Jun and Zhang, Min}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {157593--157623}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26ed/zhang26ed.pdf}, url = {https://proceedings.mlr.press/v306/zhang26ed.html}, abstract = {Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit modality preference, a tendency to favor one modality over another when processing multi-modal contexts. To study this question, we introduce $MC^2$ benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making. Extensive experiments reveal that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences are statistically associated with performances of downstream taks for MLLMs. Further analysis shows that modality preference can be controlled by instruction guidance and captured within the latent representations of MLLMs. Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning. This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple multi-modal understanding and reasoning tasks.} }
Endnote
%0 Conference Paper %T Evaluating and Steering Modality Preferences in Multi-modal LLMs %A Yu Zhang %A Jinlong Ma %A Yongshuai Hou %A Xuefeng Bai %A Kehai Chen %A Yang Xiang %A Jun Yu %A Min Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26ed %I PMLR %P 157593--157623 %U https://proceedings.mlr.press/v306/zhang26ed.html %V 306 %X Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit modality preference, a tendency to favor one modality over another when processing multi-modal contexts. To study this question, we introduce $MC^2$ benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making. Extensive experiments reveal that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences are statistically associated with performances of downstream taks for MLLMs. Further analysis shows that modality preference can be controlled by instruction guidance and captured within the latent representations of MLLMs. Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning. This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple multi-modal understanding and reasoning tasks.
APA
Zhang, Y., Ma, J., Hou, Y., Bai, X., Chen, K., Xiang, Y., Yu, J. & Zhang, M.. (2026). Evaluating and Steering Modality Preferences in Multi-modal LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:157593-157623 Available from https://proceedings.mlr.press/v306/zhang26ed.html.

Related Material