Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models

Liangchen Liu, Nannan Wang, Xi Yang, Xinbo Gao, Tongliang Liu
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:39755-39773, 2025.

Abstract

Prompt learning is a cutting-edge parameter-efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learning a single text prompt, recent works have revealed that learning diverse text prompts can effectively boost the performances on downstream tasks, as the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational resources. This efficiency issue still remains unexplored. To achieve efficient and diverse prompt learning, this paper proposes a novel Surrogate Prompt Learning (SurPL) framework. Instead of learning diverse text prompts, SurPL directly generates the desired prompted text features via a lightweight Surrogate Feature Generator (SFG), thereby avoiding the complex gradient computation procedure of conventional diverse prompt learning. Concretely, based on a basic prompted text feature, SFG can directly and efficiently generate diverse prompted features according to different pre-defined conditional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL on various benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-liu25bx, title = {Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models}, author = {Liu, Liangchen and Wang, Nannan and Yang, Xi and Gao, Xinbo and Liu, Tongliang}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {39755--39773}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/liu25bx/liu25bx.pdf}, url = {https://proceedings.mlr.press/v267/liu25bx.html}, abstract = {Prompt learning is a cutting-edge parameter-efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learning a single text prompt, recent works have revealed that learning diverse text prompts can effectively boost the performances on downstream tasks, as the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational resources. This efficiency issue still remains unexplored. To achieve efficient and diverse prompt learning, this paper proposes a novel Surrogate Prompt Learning (SurPL) framework. Instead of learning diverse text prompts, SurPL directly generates the desired prompted text features via a lightweight Surrogate Feature Generator (SFG), thereby avoiding the complex gradient computation procedure of conventional diverse prompt learning. Concretely, based on a basic prompted text feature, SFG can directly and efficiently generate diverse prompted features according to different pre-defined conditional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL on various benchmarks.} }
Endnote
%0 Conference Paper %T Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models %A Liangchen Liu %A Nannan Wang %A Xi Yang %A Xinbo Gao %A Tongliang Liu %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-liu25bx %I PMLR %P 39755--39773 %U https://proceedings.mlr.press/v267/liu25bx.html %V 267 %X Prompt learning is a cutting-edge parameter-efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learning a single text prompt, recent works have revealed that learning diverse text prompts can effectively boost the performances on downstream tasks, as the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational resources. This efficiency issue still remains unexplored. To achieve efficient and diverse prompt learning, this paper proposes a novel Surrogate Prompt Learning (SurPL) framework. Instead of learning diverse text prompts, SurPL directly generates the desired prompted text features via a lightweight Surrogate Feature Generator (SFG), thereby avoiding the complex gradient computation procedure of conventional diverse prompt learning. Concretely, based on a basic prompted text feature, SFG can directly and efficiently generate diverse prompted features according to different pre-defined conditional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL on various benchmarks.
APA
Liu, L., Wang, N., Yang, X., Gao, X. & Liu, T.. (2025). Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:39755-39773 Available from https://proceedings.mlr.press/v267/liu25bx.html.

Related Material