Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation

Neeraj Gangwar, Anshuka Rangi, Rishabh Deshmukh, Holakou Rahmanian, Yesh Dattatreya, Nickvash Kani
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:3916-3924, 2026.

Abstract

Parameter-efficient fine-tuning methods have emerged as a promising solution for adapting pre-trained models to various downstream tasks. While these methods perform well in single-task learning, extending them to multi-task learning exacerbates common issues, such as task interference and negative transfer, due to the limited number of trainable parameters. To address these challenges, we introduce progressive task-specific multi-task adaptation, a novel parameter-efficient approach for multi-task learning. Our approach introduces adapter modules that are shared in early layers and become increasingly task-specific in later layers. Additionally, we propose a gradient-based approach for computing task similarity and use this measure to allocate similar tasks to the shared adapter modules. To evaluate our approach, we adapt Swin and Pyramid Vision Transformers on PASCAL and NYUD-v2. On both datasets, our approach outperforms prior parameter-efficient multi-task methods while using fewer trainable parameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-gangwar26a, title = { Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation }, author = {Gangwar, Neeraj and Rangi, Anshuka and Deshmukh, Rishabh and Rahmanian, Holakou and Dattatreya, Yesh and Kani, Nickvash}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {3916--3924}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/gangwar26a/gangwar26a.pdf}, url = {https://proceedings.mlr.press/v300/gangwar26a.html}, abstract = { Parameter-efficient fine-tuning methods have emerged as a promising solution for adapting pre-trained models to various downstream tasks. While these methods perform well in single-task learning, extending them to multi-task learning exacerbates common issues, such as task interference and negative transfer, due to the limited number of trainable parameters. To address these challenges, we introduce progressive task-specific multi-task adaptation, a novel parameter-efficient approach for multi-task learning. Our approach introduces adapter modules that are shared in early layers and become increasingly task-specific in later layers. Additionally, we propose a gradient-based approach for computing task similarity and use this measure to allocate similar tasks to the shared adapter modules. To evaluate our approach, we adapt Swin and Pyramid Vision Transformers on PASCAL and NYUD-v2. On both datasets, our approach outperforms prior parameter-efficient multi-task methods while using fewer trainable parameters. } }
Endnote
%0 Conference Paper %T Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation %A Neeraj Gangwar %A Anshuka Rangi %A Rishabh Deshmukh %A Holakou Rahmanian %A Yesh Dattatreya %A Nickvash Kani %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-gangwar26a %I PMLR %P 3916--3924 %U https://proceedings.mlr.press/v300/gangwar26a.html %V 300 %X Parameter-efficient fine-tuning methods have emerged as a promising solution for adapting pre-trained models to various downstream tasks. While these methods perform well in single-task learning, extending them to multi-task learning exacerbates common issues, such as task interference and negative transfer, due to the limited number of trainable parameters. To address these challenges, we introduce progressive task-specific multi-task adaptation, a novel parameter-efficient approach for multi-task learning. Our approach introduces adapter modules that are shared in early layers and become increasingly task-specific in later layers. Additionally, we propose a gradient-based approach for computing task similarity and use this measure to allocate similar tasks to the shared adapter modules. To evaluate our approach, we adapt Swin and Pyramid Vision Transformers on PASCAL and NYUD-v2. On both datasets, our approach outperforms prior parameter-efficient multi-task methods while using fewer trainable parameters.
APA
Gangwar, N., Rangi, A., Deshmukh, R., Rahmanian, H., Dattatreya, Y. & Kani, N.. (2026). Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:3916-3924 Available from https://proceedings.mlr.press/v300/gangwar26a.html.

Related Material