I2I: Initializing Adapters with Improvised Knowledge

Tejas Srinivasan, Furong Jia, Mohammad Rostami, Jesse Thomason
Proceedings of The 2nd Conference on Lifelong Learning Agents, PMLR 232:923-935, 2023.

Abstract

Adapters present a promising solution to the catastrophic forgetting problem in continual learning. However, training independent Adapter modules for every new task misses an opportunity for cross-task knowledge transfer. We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks’ Adapters. We evaluate I2I on CLiMB, a multimodal continual learning benchmark, by conducting experiments on sequences of visual question answering tasks. Adapters trained with I2I consistently achieve better task accuracy than independently-trained Adapters, demonstrating that our algorithm facilitates knowledge transfer between task Adapters. I2I also results in better cross-task knowledge transfer than the state-of-the-art AdapterFusion without incurring the associated parametric cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v232-srinivasan23a, title = {I2I: Initializing Adapters with Improvised Knowledge}, author = {Srinivasan, Tejas and Jia, Furong and Rostami, Mohammad and Thomason, Jesse}, booktitle = {Proceedings of The 2nd Conference on Lifelong Learning Agents}, pages = {923--935}, year = {2023}, editor = {Chandar, Sarath and Pascanu, Razvan and Sedghi, Hanie and Precup, Doina}, volume = {232}, series = {Proceedings of Machine Learning Research}, month = {22--25 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v232/srinivasan23a/srinivasan23a.pdf}, url = {https://proceedings.mlr.press/v232/srinivasan23a.html}, abstract = {Adapters present a promising solution to the catastrophic forgetting problem in continual learning. However, training independent Adapter modules for every new task misses an opportunity for cross-task knowledge transfer. We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks’ Adapters. We evaluate I2I on CLiMB, a multimodal continual learning benchmark, by conducting experiments on sequences of visual question answering tasks. Adapters trained with I2I consistently achieve better task accuracy than independently-trained Adapters, demonstrating that our algorithm facilitates knowledge transfer between task Adapters. I2I also results in better cross-task knowledge transfer than the state-of-the-art AdapterFusion without incurring the associated parametric cost.} }
Endnote
%0 Conference Paper %T I2I: Initializing Adapters with Improvised Knowledge %A Tejas Srinivasan %A Furong Jia %A Mohammad Rostami %A Jesse Thomason %B Proceedings of The 2nd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2023 %E Sarath Chandar %E Razvan Pascanu %E Hanie Sedghi %E Doina Precup %F pmlr-v232-srinivasan23a %I PMLR %P 923--935 %U https://proceedings.mlr.press/v232/srinivasan23a.html %V 232 %X Adapters present a promising solution to the catastrophic forgetting problem in continual learning. However, training independent Adapter modules for every new task misses an opportunity for cross-task knowledge transfer. We propose Improvise to Initialize (I2I), a continual learning algorithm that initializes Adapters for incoming tasks by distilling knowledge from previously-learned tasks’ Adapters. We evaluate I2I on CLiMB, a multimodal continual learning benchmark, by conducting experiments on sequences of visual question answering tasks. Adapters trained with I2I consistently achieve better task accuracy than independently-trained Adapters, demonstrating that our algorithm facilitates knowledge transfer between task Adapters. I2I also results in better cross-task knowledge transfer than the state-of-the-art AdapterFusion without incurring the associated parametric cost.
APA
Srinivasan, T., Jia, F., Rostami, M. & Thomason, J.. (2023). I2I: Initializing Adapters with Improvised Knowledge. Proceedings of The 2nd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 232:923-935 Available from https://proceedings.mlr.press/v232/srinivasan23a.html.

Related Material