Exploiting Presentative Feature Distributions for Parameter-Efficient Continual Learning of Large Language Models

Xin Cheng, Jiabo Ye, Haiyang Xu, Ming Yan, Ji Zhang, Feng Liu, Fei Huang, Lei Feng
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:10159-10181, 2025.

Abstract

Endowing large language models (LLMs) with continual learning (CL) capacities is practically important, which enables them to dynamically acquire new knowledge over time. Although many effective methods have been proposed for CL of LLMs, they did not consider online scenarios, thereby sharing a common problem: information leakage (IL), where the task-related information of learned tasks is accessed or reused again. IL not only imposes potential risks on data privacy protection but also significantly hinders the deployment of LLMs in real-world scenarios. To avoid IL while maintaining outstanding CL performance, we propose a novel CL method for LLMs, which first characterizes a parameter-efficient fine-tuning (PEFT) block by a presentative feature distribution, and then dynamically selects the appropriate PEFT blocks for each instance based on its similarity with the presentative feature distributions. Extensive experiments validate the effectiveness of our method on the CL of LLM, showcasing its potential to enhance both privacy and adaptability in practical applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-cheng25j, title = {Exploiting Presentative Feature Distributions for Parameter-Efficient Continual Learning of Large Language Models}, author = {Cheng, Xin and Ye, Jiabo and Xu, Haiyang and Yan, Ming and Zhang, Ji and Liu, Feng and Huang, Fei and Feng, Lei}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {10159--10181}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/cheng25j/cheng25j.pdf}, url = {https://proceedings.mlr.press/v267/cheng25j.html}, abstract = {Endowing large language models (LLMs) with continual learning (CL) capacities is practically important, which enables them to dynamically acquire new knowledge over time. Although many effective methods have been proposed for CL of LLMs, they did not consider online scenarios, thereby sharing a common problem: information leakage (IL), where the task-related information of learned tasks is accessed or reused again. IL not only imposes potential risks on data privacy protection but also significantly hinders the deployment of LLMs in real-world scenarios. To avoid IL while maintaining outstanding CL performance, we propose a novel CL method for LLMs, which first characterizes a parameter-efficient fine-tuning (PEFT) block by a presentative feature distribution, and then dynamically selects the appropriate PEFT blocks for each instance based on its similarity with the presentative feature distributions. Extensive experiments validate the effectiveness of our method on the CL of LLM, showcasing its potential to enhance both privacy and adaptability in practical applications.} }
Endnote
%0 Conference Paper %T Exploiting Presentative Feature Distributions for Parameter-Efficient Continual Learning of Large Language Models %A Xin Cheng %A Jiabo Ye %A Haiyang Xu %A Ming Yan %A Ji Zhang %A Feng Liu %A Fei Huang %A Lei Feng %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-cheng25j %I PMLR %P 10159--10181 %U https://proceedings.mlr.press/v267/cheng25j.html %V 267 %X Endowing large language models (LLMs) with continual learning (CL) capacities is practically important, which enables them to dynamically acquire new knowledge over time. Although many effective methods have been proposed for CL of LLMs, they did not consider online scenarios, thereby sharing a common problem: information leakage (IL), where the task-related information of learned tasks is accessed or reused again. IL not only imposes potential risks on data privacy protection but also significantly hinders the deployment of LLMs in real-world scenarios. To avoid IL while maintaining outstanding CL performance, we propose a novel CL method for LLMs, which first characterizes a parameter-efficient fine-tuning (PEFT) block by a presentative feature distribution, and then dynamically selects the appropriate PEFT blocks for each instance based on its similarity with the presentative feature distributions. Extensive experiments validate the effectiveness of our method on the CL of LLM, showcasing its potential to enhance both privacy and adaptability in practical applications.
APA
Cheng, X., Ye, J., Xu, H., Yan, M., Zhang, J., Liu, F., Huang, F. & Feng, L.. (2025). Exploiting Presentative Feature Distributions for Parameter-Efficient Continual Learning of Large Language Models. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:10159-10181 Available from https://proceedings.mlr.press/v267/cheng25j.html.

Related Material