DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models

Hyein Seo, Yuna Jeong, Mingyu Kang, Junhyeong Park, Yong Suk Choi
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:109025-109044, 2026.

Abstract

Large Vision–Language Models (LVLMs) have shown outstanding performance across various multimodal tasks, but they still suffer from hallucinations, where they generate incorrect information by relying on language priors without visual grounding. To alleviate this issue, prior work has explored contrastive decoding approaches that compare the output of an original LVLM with that of a contrast branch. However, existing methods typically generate contrast logits through preprocessing of the input image. Such input-level perturbations fail to sufficiently reflect the model’s internal degree of visual reliance during the decoding process. To address this limitation, we propose Dynamic Entropy-Guided Attention Perturbation (DEGAP) for contrastive decoding in LVLMs. DEGAP performs contrastive decoding by directly perturbing visual attention and leveraging the resulting logits, without requiring any additional image preprocessing. To this end, we analyze the layer-wise effects of visual attention perturbations and, based on these observations, dynamically select the layers at which attention perturbation is applied according to the model’s confidence. Experimental results on seven benchmarks demonstrate that DEGAP effectively mitigates various types of hallucinations and consistently outperforms state-of-the-art methods in general VQA performance.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-seo26b, title = {{DEGAP}: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models}, author = {Seo, Hyein and Jeong, Yuna and Kang, Mingyu and Park, Junhyeong and Choi, Yong Suk}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {109025--109044}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/seo26b/seo26b.pdf}, url = {https://proceedings.mlr.press/v306/seo26b.html}, abstract = {Large Vision–Language Models (LVLMs) have shown outstanding performance across various multimodal tasks, but they still suffer from hallucinations, where they generate incorrect information by relying on language priors without visual grounding. To alleviate this issue, prior work has explored contrastive decoding approaches that compare the output of an original LVLM with that of a contrast branch. However, existing methods typically generate contrast logits through preprocessing of the input image. Such input-level perturbations fail to sufficiently reflect the model’s internal degree of visual reliance during the decoding process. To address this limitation, we propose Dynamic Entropy-Guided Attention Perturbation (DEGAP) for contrastive decoding in LVLMs. DEGAP performs contrastive decoding by directly perturbing visual attention and leveraging the resulting logits, without requiring any additional image preprocessing. To this end, we analyze the layer-wise effects of visual attention perturbations and, based on these observations, dynamically select the layers at which attention perturbation is applied according to the model’s confidence. Experimental results on seven benchmarks demonstrate that DEGAP effectively mitigates various types of hallucinations and consistently outperforms state-of-the-art methods in general VQA performance.} }
Endnote
%0 Conference Paper %T DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models %A Hyein Seo %A Yuna Jeong %A Mingyu Kang %A Junhyeong Park %A Yong Suk Choi %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-seo26b %I PMLR %P 109025--109044 %U https://proceedings.mlr.press/v306/seo26b.html %V 306 %X Large Vision–Language Models (LVLMs) have shown outstanding performance across various multimodal tasks, but they still suffer from hallucinations, where they generate incorrect information by relying on language priors without visual grounding. To alleviate this issue, prior work has explored contrastive decoding approaches that compare the output of an original LVLM with that of a contrast branch. However, existing methods typically generate contrast logits through preprocessing of the input image. Such input-level perturbations fail to sufficiently reflect the model’s internal degree of visual reliance during the decoding process. To address this limitation, we propose Dynamic Entropy-Guided Attention Perturbation (DEGAP) for contrastive decoding in LVLMs. DEGAP performs contrastive decoding by directly perturbing visual attention and leveraging the resulting logits, without requiring any additional image preprocessing. To this end, we analyze the layer-wise effects of visual attention perturbations and, based on these observations, dynamically select the layers at which attention perturbation is applied according to the model’s confidence. Experimental results on seven benchmarks demonstrate that DEGAP effectively mitigates various types of hallucinations and consistently outperforms state-of-the-art methods in general VQA performance.
APA
Seo, H., Jeong, Y., Kang, M., Park, J. & Choi, Y.S.. (2026). DEGAP: Dynamic Entropy-Guided Attention Perturbation for Contrastive Decoding in Large Vision-Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:109025-109044 Available from https://proceedings.mlr.press/v306/seo26b.html.

Related Material