Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:18828-18860, 2026.

Abstract

Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-cheng26d, title = {Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating}, author = {Cheng, Zhe and Chen, Wenyu and Zhang, Fode and Shen, Dehuan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {18828--18860}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/cheng26d/cheng26d.pdf}, url = {https://proceedings.mlr.press/v306/cheng26d.html}, abstract = {Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.} }
Endnote
%0 Conference Paper %T Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating %A Zhe Cheng %A Wenyu Chen %A Fode Zhang %A Dehuan Shen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-cheng26d %I PMLR %P 18828--18860 %U https://proceedings.mlr.press/v306/cheng26d.html %V 306 %X Large vision-language models (LVLMs) often hallucinate content that is fluent yet unsupported by the image, limiting their reliability in real-world deployment. We show that a key failure mode arises from route competition: even when visual tokens receive attention, the final token decision can be dominated by the textual pathway, causing the decoder to follow linguistic priors over visual evidence. To mitigate this, we propose a training-free, decision-aligned intervention that decomposes each attention head into a visual route and a text route, and estimates their token-level effects using an efficient one-forward/one-gradient approximation. These estimates reveal route conflict within heads and identify prior-dominant ones, enabling selective suppression of only the text route while keeping the visual route intact. Across five benchmarks spanning discriminative and generative settings, our method consistently reduces hallucination-related errors across models with limited impact on overall multimodal performance, while incurring a modest inference-time overhead.
APA
Cheng, Z., Chen, W., Zhang, F. & Shen, D.. (2026). Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:18828-18860 Available from https://proceedings.mlr.press/v306/cheng26d.html.

Related Material