Controllable Prompt Tuning For Balancing Group Distributional Robustness

Hoang Phan, Andrew Gordon Wilson, Qi Lei
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:40667-40687, 2024.

Abstract

Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts. While recent methods have largely focused on optimizing the worst-group objective, this often comes at the expense of good performance on other groups. To address this problem, we introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. However, directly applying such optimization involves updating the parameters of the entire network, making it both computationally expensive and challenging. Thus, we introduce Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data, while requiring only $0.4%$ tunable parameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-phan24b, title = {Controllable Prompt Tuning For Balancing Group Distributional Robustness}, author = {Phan, Hoang and Wilson, Andrew Gordon and Lei, Qi}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {40667--40687}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/phan24b/phan24b.pdf}, url = {https://proceedings.mlr.press/v235/phan24b.html}, abstract = {Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts. While recent methods have largely focused on optimizing the worst-group objective, this often comes at the expense of good performance on other groups. To address this problem, we introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. However, directly applying such optimization involves updating the parameters of the entire network, making it both computationally expensive and challenging. Thus, we introduce Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data, while requiring only $0.4%$ tunable parameters.} }
Endnote
%0 Conference Paper %T Controllable Prompt Tuning For Balancing Group Distributional Robustness %A Hoang Phan %A Andrew Gordon Wilson %A Qi Lei %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-phan24b %I PMLR %P 40667--40687 %U https://proceedings.mlr.press/v235/phan24b.html %V 235 %X Models trained on data composed of different groups or domains can suffer from severe performance degradation under distribution shifts. While recent methods have largely focused on optimizing the worst-group objective, this often comes at the expense of good performance on other groups. To address this problem, we introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. However, directly applying such optimization involves updating the parameters of the entire network, making it both computationally expensive and challenging. Thus, we introduce Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data, while requiring only $0.4%$ tunable parameters.
APA
Phan, H., Wilson, A.G. & Lei, Q.. (2024). Controllable Prompt Tuning For Balancing Group Distributional Robustness. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:40667-40687 Available from https://proceedings.mlr.press/v235/phan24b.html.

Related Material