SPR: A Structured Prompt Refinement Network for Modality Missing

Hao Chen, Diwei Su, Zhuo Wang, Zuwang He, Menglu Chen, Xiuxing Li, Xia Wu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:16657-16671, 2026.

Abstract

Prompt learning has recently emerged as a dominant paradigm to tackle the missing modalities challenge. However, existing methods often overlook the internal structural information of prompt vectors, limiting performance in guiding frozen backbone models under diverse missing modality scenarios. To address this, we propose a Structured Prompt Refining (SPR) network that refines the internal structure of prompt vectors across multiple dimensions: (1) a Global Interaction Fusion Module captures bidirectional interactions across prompt layers, thereby mitigating sub-optimal adaptation from inconsistent guidance under missing modalities, (2) a Local Feature Refinement Module structures adjacent prompt vectors into coherent semantic units, leveraging local contextual relationships to maintain semantic integrity during modality absence, and (3) a Channel Feature Selection Module uses point-wise gating to adaptively suppress noise and enhance critical channels based on the specific missing modality. Using only 0.8% trainable parameters, SPR achieves significant improvements on three mainstream multimodal classification datasets. Notably, it surpasses state-of-the-art by 3.8% in F1-Macro on the MM-IMDB dataset, even at a 90% modality missing rate. Extensive experiments and in-depth ablations validate SPR’s effectiveness and robustness under various missing conditions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26dv, title = {{SPR}: A Structured Prompt Refinement Network for Modality Missing}, author = {Chen, Hao and Su, Diwei and Wang, Zhuo and He, Zuwang and Chen, Menglu and Li, Xiuxing and Wu, Xia}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {16657--16671}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26dv/chen26dv.pdf}, url = {https://proceedings.mlr.press/v306/chen26dv.html}, abstract = {Prompt learning has recently emerged as a dominant paradigm to tackle the missing modalities challenge. However, existing methods often overlook the internal structural information of prompt vectors, limiting performance in guiding frozen backbone models under diverse missing modality scenarios. To address this, we propose a Structured Prompt Refining (SPR) network that refines the internal structure of prompt vectors across multiple dimensions: (1) a Global Interaction Fusion Module captures bidirectional interactions across prompt layers, thereby mitigating sub-optimal adaptation from inconsistent guidance under missing modalities, (2) a Local Feature Refinement Module structures adjacent prompt vectors into coherent semantic units, leveraging local contextual relationships to maintain semantic integrity during modality absence, and (3) a Channel Feature Selection Module uses point-wise gating to adaptively suppress noise and enhance critical channels based on the specific missing modality. Using only 0.8% trainable parameters, SPR achieves significant improvements on three mainstream multimodal classification datasets. Notably, it surpasses state-of-the-art by 3.8% in F1-Macro on the MM-IMDB dataset, even at a 90% modality missing rate. Extensive experiments and in-depth ablations validate SPR’s effectiveness and robustness under various missing conditions.} }
Endnote
%0 Conference Paper %T SPR: A Structured Prompt Refinement Network for Modality Missing %A Hao Chen %A Diwei Su %A Zhuo Wang %A Zuwang He %A Menglu Chen %A Xiuxing Li %A Xia Wu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26dv %I PMLR %P 16657--16671 %U https://proceedings.mlr.press/v306/chen26dv.html %V 306 %X Prompt learning has recently emerged as a dominant paradigm to tackle the missing modalities challenge. However, existing methods often overlook the internal structural information of prompt vectors, limiting performance in guiding frozen backbone models under diverse missing modality scenarios. To address this, we propose a Structured Prompt Refining (SPR) network that refines the internal structure of prompt vectors across multiple dimensions: (1) a Global Interaction Fusion Module captures bidirectional interactions across prompt layers, thereby mitigating sub-optimal adaptation from inconsistent guidance under missing modalities, (2) a Local Feature Refinement Module structures adjacent prompt vectors into coherent semantic units, leveraging local contextual relationships to maintain semantic integrity during modality absence, and (3) a Channel Feature Selection Module uses point-wise gating to adaptively suppress noise and enhance critical channels based on the specific missing modality. Using only 0.8% trainable parameters, SPR achieves significant improvements on three mainstream multimodal classification datasets. Notably, it surpasses state-of-the-art by 3.8% in F1-Macro on the MM-IMDB dataset, even at a 90% modality missing rate. Extensive experiments and in-depth ablations validate SPR’s effectiveness and robustness under various missing conditions.
APA
Chen, H., Su, D., Wang, Z., He, Z., Chen, M., Li, X. & Wu, X.. (2026). SPR: A Structured Prompt Refinement Network for Modality Missing. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:16657-16671 Available from https://proceedings.mlr.press/v306/chen26dv.html.

Related Material