Unified Confidence Adjustment for Robust Cross-Modal Retrieval under Test-Time Distribution Shifts

Rui Zhou, Yawen Hao, Hao Zuo, Xinhang Wan, Cheng Zhu, Yun Zhou
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:8292-8311, 2026.

Abstract

Cross-modal retrieval models often suffer substantial performance degradation under test-time distribution shifts. Existing test-time adaptation methods primarily based on entropy minimization tend to sharpen the similarity distribution. However, retrieval relies on feature similarities to rank candidates, and the similarity gaps between nearby-ranked candidates are often small. Over-sharpening the similarity gap between the candidates can distort the semantic similarity structure, leading to miscalibrated updates and unstable retrieval. To address this issue, we propose a novel *Unified Confidence Adjustment* framework that explicitly accounts for semantic similarity within the candidate set. Specifically, we propose the *Semantic Proximity Margin* as a confidence prior. Building on this margin, we develop a *Confidence State Identification* mechanism and a *Unified Confidence Adjustment* strategy for adaptive confidence calibration. Extensive experiments on four benchmarks under both zero-shot transfer and natural corruption settings show that our proposal consistently outperforms state-of-the-art test-time adaptation methods with minimal computational overhead, underscoring the effectiveness of confidence-aware regularization for robust cross-modal retrieval.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-zhou26b, title = {Unified Confidence Adjustment for Robust Cross-Modal Retrieval under Test-Time Distribution Shifts}, author = {Zhou, Rui and Hao, Yawen and Zuo, Hao and Wan, Xinhang and Zhu, Cheng and Zhou, Yun}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {8292--8311}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/zhou26b/zhou26b.pdf}, url = {https://proceedings.mlr.press/v337/zhou26b.html}, abstract = {Cross-modal retrieval models often suffer substantial performance degradation under test-time distribution shifts. Existing test-time adaptation methods primarily based on entropy minimization tend to sharpen the similarity distribution. However, retrieval relies on feature similarities to rank candidates, and the similarity gaps between nearby-ranked candidates are often small. Over-sharpening the similarity gap between the candidates can distort the semantic similarity structure, leading to miscalibrated updates and unstable retrieval. To address this issue, we propose a novel *Unified Confidence Adjustment* framework that explicitly accounts for semantic similarity within the candidate set. Specifically, we propose the *Semantic Proximity Margin* as a confidence prior. Building on this margin, we develop a *Confidence State Identification* mechanism and a *Unified Confidence Adjustment* strategy for adaptive confidence calibration. Extensive experiments on four benchmarks under both zero-shot transfer and natural corruption settings show that our proposal consistently outperforms state-of-the-art test-time adaptation methods with minimal computational overhead, underscoring the effectiveness of confidence-aware regularization for robust cross-modal retrieval.} }
Endnote
%0 Conference Paper %T Unified Confidence Adjustment for Robust Cross-Modal Retrieval under Test-Time Distribution Shifts %A Rui Zhou %A Yawen Hao %A Hao Zuo %A Xinhang Wan %A Cheng Zhu %A Yun Zhou %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-zhou26b %I PMLR %P 8292--8311 %U https://proceedings.mlr.press/v337/zhou26b.html %V 337 %X Cross-modal retrieval models often suffer substantial performance degradation under test-time distribution shifts. Existing test-time adaptation methods primarily based on entropy minimization tend to sharpen the similarity distribution. However, retrieval relies on feature similarities to rank candidates, and the similarity gaps between nearby-ranked candidates are often small. Over-sharpening the similarity gap between the candidates can distort the semantic similarity structure, leading to miscalibrated updates and unstable retrieval. To address this issue, we propose a novel *Unified Confidence Adjustment* framework that explicitly accounts for semantic similarity within the candidate set. Specifically, we propose the *Semantic Proximity Margin* as a confidence prior. Building on this margin, we develop a *Confidence State Identification* mechanism and a *Unified Confidence Adjustment* strategy for adaptive confidence calibration. Extensive experiments on four benchmarks under both zero-shot transfer and natural corruption settings show that our proposal consistently outperforms state-of-the-art test-time adaptation methods with minimal computational overhead, underscoring the effectiveness of confidence-aware regularization for robust cross-modal retrieval.
APA
Zhou, R., Hao, Y., Zuo, H., Wan, X., Zhu, C. & Zhou, Y.. (2026). Unified Confidence Adjustment for Robust Cross-Modal Retrieval under Test-Time Distribution Shifts. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:8292-8311 Available from https://proceedings.mlr.press/v337/zhou26b.html.

Related Material