Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data

Mingcai Chen, Baoming Zhang, Zongbo Han, Wenyu Jiang, Yanmeng Wang, Shuai Feng, Yuntao Du., Bingkun Bao
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:9711-9727, 2025.

Abstract

Modern machine learning applications are characterized by the increasing size of deep models and the growing diversity of data modalities. This trend underscores the importance of efficiently adapting pre-trained multi-modal models to the test distribution in real time, i.e., multi-modal test-time adaptation. In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners. In this research, we investigate the the under-explored practical scenario uni-modal distribution shift, where the distribution shift influences only one modality, leaving the others unchanged. Through theoretical and empirical analyses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques. To flexibly combat this unique shift, we propose a selective adaptation schema that incorporates multiple modality-specific adapters to accommodate potential shifts and a “router” module that determines which modality requires adaptation. Finally, we validate the effectiveness of our proposed method through extensive experimental evaluations. Code available at https://github.com/chenmc1996/Uni-Modal-Distribution-Shift.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-chen25ch, title = {Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data}, author = {Chen, Mingcai and Zhang, Baoming and Han, Zongbo and Jiang, Wenyu and Wang, Yanmeng and Feng, Shuai and Du., Yuntao and Bao, Bingkun}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {9711--9727}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/chen25ch/chen25ch.pdf}, url = {https://proceedings.mlr.press/v267/chen25ch.html}, abstract = {Modern machine learning applications are characterized by the increasing size of deep models and the growing diversity of data modalities. This trend underscores the importance of efficiently adapting pre-trained multi-modal models to the test distribution in real time, i.e., multi-modal test-time adaptation. In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners. In this research, we investigate the the under-explored practical scenario uni-modal distribution shift, where the distribution shift influences only one modality, leaving the others unchanged. Through theoretical and empirical analyses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques. To flexibly combat this unique shift, we propose a selective adaptation schema that incorporates multiple modality-specific adapters to accommodate potential shifts and a “router” module that determines which modality requires adaptation. Finally, we validate the effectiveness of our proposed method through extensive experimental evaluations. Code available at https://github.com/chenmc1996/Uni-Modal-Distribution-Shift.} }
Endnote
%0 Conference Paper %T Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data %A Mingcai Chen %A Baoming Zhang %A Zongbo Han %A Wenyu Jiang %A Yanmeng Wang %A Shuai Feng %A Yuntao Du. %A Bingkun Bao %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-chen25ch %I PMLR %P 9711--9727 %U https://proceedings.mlr.press/v267/chen25ch.html %V 267 %X Modern machine learning applications are characterized by the increasing size of deep models and the growing diversity of data modalities. This trend underscores the importance of efficiently adapting pre-trained multi-modal models to the test distribution in real time, i.e., multi-modal test-time adaptation. In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners. In this research, we investigate the the under-explored practical scenario uni-modal distribution shift, where the distribution shift influences only one modality, leaving the others unchanged. Through theoretical and empirical analyses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques. To flexibly combat this unique shift, we propose a selective adaptation schema that incorporates multiple modality-specific adapters to accommodate potential shifts and a “router” module that determines which modality requires adaptation. Finally, we validate the effectiveness of our proposed method through extensive experimental evaluations. Code available at https://github.com/chenmc1996/Uni-Modal-Distribution-Shift.
APA
Chen, M., Zhang, B., Han, Z., Jiang, W., Wang, Y., Feng, S., Du., Y. & Bao, B.. (2025). Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:9711-9727 Available from https://proceedings.mlr.press/v267/chen25ch.html.

Related Material