MADA-Attack: Transferable Multi-modal Attention Distraction Adversarial Attack against Vision Language Models

Zhihan Qin, Jiahao Chen, Chunyi Zhou, Yuwen Pu, Chunqiang Hu, Xiaolei Liu, Shouling Ji
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:101524-101546, 2026.

Abstract

Vision Language Models (VLMs) achieve strong performance across multi-modal tasks but remain vulnerable to universal adversarial perturbations (UAPs). Existing UAP methods mainly operate on the visual modality, overlooking structured textual semantics and cross-modal interactions, which limits their ability to disrupt alignment and generalize across tasks and model architectures. To address these limits, we propose Multi-modal Attention Distraction Adversarial Attack (MADA-Attack) framework. We begin by conducting several insight experiments and discover that modality attention distributes differently over layers and early phase of optimization is decisive. Building on these observations, we introduce Semantic Token Manipulation (STM) to steer text-guided attention, and Fused Embedding Training (FET) to jointly optimize textual and visual embedding losses for coordinated misalignment. We further incorporate an Adaptive Data Augmentation (ADA) strategy that dynamically balances attack strength, transferability, and training efficiency. Extensive experiments demonstrate that MADA-Attack consistently achieves state-of-the-art performance and strong transferability while remaining computationally lightweight, with an average ASR of 82.60% and 73.42% in zero-shot classification and image captioning tasks. For the visual question answering (VQA) and I-T Retrieval task, our method exceeds the SOTA baseline by 10%. Our code is available at this GitHub Repository.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-qin26a, title = {{MADA}-Attack: Transferable Multi-modal Attention Distraction Adversarial Attack against Vision Language Models}, author = {Qin, Zhihan and Chen, Jiahao and Zhou, Chunyi and Pu, Yuwen and Hu, Chunqiang and Liu, Xiaolei and Ji, Shouling}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {101524--101546}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/qin26a/qin26a.pdf}, url = {https://proceedings.mlr.press/v306/qin26a.html}, abstract = {Vision Language Models (VLMs) achieve strong performance across multi-modal tasks but remain vulnerable to universal adversarial perturbations (UAPs). Existing UAP methods mainly operate on the visual modality, overlooking structured textual semantics and cross-modal interactions, which limits their ability to disrupt alignment and generalize across tasks and model architectures. To address these limits, we propose Multi-modal Attention Distraction Adversarial Attack (MADA-Attack) framework. We begin by conducting several insight experiments and discover that modality attention distributes differently over layers and early phase of optimization is decisive. Building on these observations, we introduce Semantic Token Manipulation (STM) to steer text-guided attention, and Fused Embedding Training (FET) to jointly optimize textual and visual embedding losses for coordinated misalignment. We further incorporate an Adaptive Data Augmentation (ADA) strategy that dynamically balances attack strength, transferability, and training efficiency. Extensive experiments demonstrate that MADA-Attack consistently achieves state-of-the-art performance and strong transferability while remaining computationally lightweight, with an average ASR of 82.60% and 73.42% in zero-shot classification and image captioning tasks. For the visual question answering (VQA) and I-T Retrieval task, our method exceeds the SOTA baseline by 10%. Our code is available at this GitHub Repository.} }
Endnote
%0 Conference Paper %T MADA-Attack: Transferable Multi-modal Attention Distraction Adversarial Attack against Vision Language Models %A Zhihan Qin %A Jiahao Chen %A Chunyi Zhou %A Yuwen Pu %A Chunqiang Hu %A Xiaolei Liu %A Shouling Ji %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-qin26a %I PMLR %P 101524--101546 %U https://proceedings.mlr.press/v306/qin26a.html %V 306 %X Vision Language Models (VLMs) achieve strong performance across multi-modal tasks but remain vulnerable to universal adversarial perturbations (UAPs). Existing UAP methods mainly operate on the visual modality, overlooking structured textual semantics and cross-modal interactions, which limits their ability to disrupt alignment and generalize across tasks and model architectures. To address these limits, we propose Multi-modal Attention Distraction Adversarial Attack (MADA-Attack) framework. We begin by conducting several insight experiments and discover that modality attention distributes differently over layers and early phase of optimization is decisive. Building on these observations, we introduce Semantic Token Manipulation (STM) to steer text-guided attention, and Fused Embedding Training (FET) to jointly optimize textual and visual embedding losses for coordinated misalignment. We further incorporate an Adaptive Data Augmentation (ADA) strategy that dynamically balances attack strength, transferability, and training efficiency. Extensive experiments demonstrate that MADA-Attack consistently achieves state-of-the-art performance and strong transferability while remaining computationally lightweight, with an average ASR of 82.60% and 73.42% in zero-shot classification and image captioning tasks. For the visual question answering (VQA) and I-T Retrieval task, our method exceeds the SOTA baseline by 10%. Our code is available at this GitHub Repository.
APA
Qin, Z., Chen, J., Zhou, C., Pu, Y., Hu, C., Liu, X. & Ji, S.. (2026). MADA-Attack: Transferable Multi-modal Attention Distraction Adversarial Attack against Vision Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:101524-101546 Available from https://proceedings.mlr.press/v306/qin26a.html.

Related Material