Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness

Xiang Fang, Wanlong Fang, Wei Ji
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:29001-29014, 2026.

Abstract

Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the "Hubris of Semantics", where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this Open-World Trustworthiness Paradox, we propose Immuno-VLM, a bio-inspired framework that adapts the biological principle of Immunological Negative Selection to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate "Semantic Antibodies", textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes. Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-fang26b, title = {Immuno-{VLM}: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness}, author = {Fang, Xiang and Fang, Wanlong and Ji, Wei}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {29001--29014}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/fang26b/fang26b.pdf}, url = {https://proceedings.mlr.press/v306/fang26b.html}, abstract = {Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the "Hubris of Semantics", where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this Open-World Trustworthiness Paradox, we propose Immuno-VLM, a bio-inspired framework that adapts the biological principle of Immunological Negative Selection to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate "Semantic Antibodies", textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes. Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.} }
Endnote
%0 Conference Paper %T Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness %A Xiang Fang %A Wanlong Fang %A Wei Ji %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-fang26b %I PMLR %P 29001--29014 %U https://proceedings.mlr.press/v306/fang26b.html %V 306 %X Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the "Hubris of Semantics", where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this Open-World Trustworthiness Paradox, we propose Immuno-VLM, a bio-inspired framework that adapts the biological principle of Immunological Negative Selection to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate "Semantic Antibodies", textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes. Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.
APA
Fang, X., Fang, W. & Ji, W.. (2026). Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:29001-29014 Available from https://proceedings.mlr.press/v306/fang26b.html.

Related Material