Federated Continual Learning via Prompt-based Dual Knowledge Transfer

Hongming Piao, Yichen Wu, Dapeng Wu, Ying Wei
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:40725-40739, 2024.

Abstract

In Federated Continual Learning (FCL), the challenge lies in effectively facilitating knowledge transfer and enhancing the performance across various tasks on different clients. Current FCL methods predominantly focus on avoiding interference between tasks, thereby overlooking the potential for positive knowledge transfer across tasks learned by different clients at separate time intervals. To address this issue, we introduce a Prompt-based knowledge transfer FCL algorithm, called Powder, designed to effectively foster the transfer of knowledge encapsulated in prompts between various sequentially learned tasks and clients. Furthermore, we have devised a unique approach for prompt generation and aggregation, intending to alleviate privacy protection concerns and communication overhead, while still promoting knowledge transfer. Comprehensive experimental results demonstrate the superiority of our method in terms of reduction in communication costs, and enhancement of knowledge transfer. Code is available at https://github.com/piaohongming/Powder.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-piao24a, title = {Federated Continual Learning via Prompt-based Dual Knowledge Transfer}, author = {Piao, Hongming and Wu, Yichen and Wu, Dapeng and Wei, Ying}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {40725--40739}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/piao24a/piao24a.pdf}, url = {https://proceedings.mlr.press/v235/piao24a.html}, abstract = {In Federated Continual Learning (FCL), the challenge lies in effectively facilitating knowledge transfer and enhancing the performance across various tasks on different clients. Current FCL methods predominantly focus on avoiding interference between tasks, thereby overlooking the potential for positive knowledge transfer across tasks learned by different clients at separate time intervals. To address this issue, we introduce a Prompt-based knowledge transfer FCL algorithm, called Powder, designed to effectively foster the transfer of knowledge encapsulated in prompts between various sequentially learned tasks and clients. Furthermore, we have devised a unique approach for prompt generation and aggregation, intending to alleviate privacy protection concerns and communication overhead, while still promoting knowledge transfer. Comprehensive experimental results demonstrate the superiority of our method in terms of reduction in communication costs, and enhancement of knowledge transfer. Code is available at https://github.com/piaohongming/Powder.} }
Endnote
%0 Conference Paper %T Federated Continual Learning via Prompt-based Dual Knowledge Transfer %A Hongming Piao %A Yichen Wu %A Dapeng Wu %A Ying Wei %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-piao24a %I PMLR %P 40725--40739 %U https://proceedings.mlr.press/v235/piao24a.html %V 235 %X In Federated Continual Learning (FCL), the challenge lies in effectively facilitating knowledge transfer and enhancing the performance across various tasks on different clients. Current FCL methods predominantly focus on avoiding interference between tasks, thereby overlooking the potential for positive knowledge transfer across tasks learned by different clients at separate time intervals. To address this issue, we introduce a Prompt-based knowledge transfer FCL algorithm, called Powder, designed to effectively foster the transfer of knowledge encapsulated in prompts between various sequentially learned tasks and clients. Furthermore, we have devised a unique approach for prompt generation and aggregation, intending to alleviate privacy protection concerns and communication overhead, while still promoting knowledge transfer. Comprehensive experimental results demonstrate the superiority of our method in terms of reduction in communication costs, and enhancement of knowledge transfer. Code is available at https://github.com/piaohongming/Powder.
APA
Piao, H., Wu, Y., Wu, D. & Wei, Y.. (2024). Federated Continual Learning via Prompt-based Dual Knowledge Transfer. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:40725-40739 Available from https://proceedings.mlr.press/v235/piao24a.html.

Related Material