Localize and Neutralize: Gradient-Guided Token Suppression Against Visual Prompt Injection Attack

Dongpeng Zhang, Ke Ma, Yangbangyan Jiang, Gaozheng Pei, Longtao Huang, Qianqian Xu, Qingming Huang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:157624-157645, 2026.

Abstract

Adversarial images pose a severe security threat to multimodal large language models through prompt injection. Existing defenses largely lack a principled understanding of the underlying mechanisms and struggle to balance efficiency and fidelity. In this work, we show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens. Based on this insight, we propose a defense that first localizes these critical tokens via gradient analysis and then neutralizes them through masking. We show that attribution based on output probabilities fails when adversarial attacks preserve the predicted token. To overcome this limitation, we introduce the Hidden-State Gradient Norm score for adversarial behavior attribution and prove that its ranking is consistent with that of the full adversarial loss gradient, providing a theoretical guarantee for accurate localization. GTM requires only a single forward–backward pass to identify and zero out a small number of high-scoring tokens, effectively disrupting the adversarial attack path. Extensive experiments on prompt injection and multimodal jailbreak attacks demonstrate that our approach reduces attack success rates (ASR) to near zero while preserving model utility with negligible computational overhead. The code is available at: https://github.com/fish883/GTM-Defense.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhang26ee, title = {Localize and Neutralize: Gradient-Guided Token Suppression Against Visual Prompt Injection Attack}, author = {Zhang, Dongpeng and Ma, Ke and Jiang, Yangbangyan and Pei, Gaozheng and Huang, Longtao and Xu, Qianqian and Huang, Qingming}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {157624--157645}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26ee/zhang26ee.pdf}, url = {https://proceedings.mlr.press/v306/zhang26ee.html}, abstract = {Adversarial images pose a severe security threat to multimodal large language models through prompt injection. Existing defenses largely lack a principled understanding of the underlying mechanisms and struggle to balance efficiency and fidelity. In this work, we show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens. Based on this insight, we propose a defense that first localizes these critical tokens via gradient analysis and then neutralizes them through masking. We show that attribution based on output probabilities fails when adversarial attacks preserve the predicted token. To overcome this limitation, we introduce the Hidden-State Gradient Norm score for adversarial behavior attribution and prove that its ranking is consistent with that of the full adversarial loss gradient, providing a theoretical guarantee for accurate localization. GTM requires only a single forward–backward pass to identify and zero out a small number of high-scoring tokens, effectively disrupting the adversarial attack path. Extensive experiments on prompt injection and multimodal jailbreak attacks demonstrate that our approach reduces attack success rates (ASR) to near zero while preserving model utility with negligible computational overhead. The code is available at: https://github.com/fish883/GTM-Defense.} }
Endnote
%0 Conference Paper %T Localize and Neutralize: Gradient-Guided Token Suppression Against Visual Prompt Injection Attack %A Dongpeng Zhang %A Ke Ma %A Yangbangyan Jiang %A Gaozheng Pei %A Longtao Huang %A Qianqian Xu %A Qingming Huang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26ee %I PMLR %P 157624--157645 %U https://proceedings.mlr.press/v306/zhang26ee.html %V 306 %X Adversarial images pose a severe security threat to multimodal large language models through prompt injection. Existing defenses largely lack a principled understanding of the underlying mechanisms and struggle to balance efficiency and fidelity. In this work, we show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens. Based on this insight, we propose a defense that first localizes these critical tokens via gradient analysis and then neutralizes them through masking. We show that attribution based on output probabilities fails when adversarial attacks preserve the predicted token. To overcome this limitation, we introduce the Hidden-State Gradient Norm score for adversarial behavior attribution and prove that its ranking is consistent with that of the full adversarial loss gradient, providing a theoretical guarantee for accurate localization. GTM requires only a single forward–backward pass to identify and zero out a small number of high-scoring tokens, effectively disrupting the adversarial attack path. Extensive experiments on prompt injection and multimodal jailbreak attacks demonstrate that our approach reduces attack success rates (ASR) to near zero while preserving model utility with negligible computational overhead. The code is available at: https://github.com/fish883/GTM-Defense.
APA
Zhang, D., Ma, K., Jiang, Y., Pei, G., Huang, L., Xu, Q. & Huang, Q.. (2026). Localize and Neutralize: Gradient-Guided Token Suppression Against Visual Prompt Injection Attack. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:157624-157645 Available from https://proceedings.mlr.press/v306/zhang26ee.html.

Related Material