AdaCL: Adaptive Continual Learning

Elif Ceren Gok Yildirim, Murat Onur Yildirim, Mert Kilickaya, Joaquin Vanschoren
Proceedings of the 1st ContinualAI Unconference, 2023, PMLR 249:15-24, 2024.

Abstract

Class-Incremental Learning aims to update a deep classifier to learn new categories while maintaining or improving its accuracy on previously observed classes. Common methods to prevent forgetting previously learned classes include regularizing the neural network updates and storing exemplars in memory, which come with hyperparameters such as the learning rate, regularization strength, or the number of exemplars. However, these hyperparameters are usually only tuned at the start and then kept fixed throughout the learning sessions, ignoring the fact that newly encountered tasks may have varying levels of novelty or difficulty. This study investigates the necessity of hyperparameter ’adaptivity’ in Class-Incremental Learning: the ability to dynamically adjust hyperparameters such as the learning rate, regularization strength, and memory size according to the properties of the new task at hand. We propose AdaCL, a Bayesian Optimization-based approach to automatically and efficiently determine the optimal values for those parameters with each learning task. We show that adapting hyperpararmeters on each new task leads to improvement in accuracy, forgetting and memory. Code is available at https://github.com/ElifCerenGokYildirim/AdaCL.

Cite this Paper

BibTeX
@InProceedings{pmlr-v249-yildirim24a, title = {AdaCL: Adaptive Continual Learning}, author = {Yildirim, Elif Ceren Gok and Yildirim, Murat Onur and Kilickaya, Mert and Vanschoren, Joaquin}, booktitle = {Proceedings of the 1st ContinualAI Unconference, 2023}, pages = {15--24}, year = {2024}, editor = {Swaroop, Siddharth and Mundt, Martin and Aljundi, Rahaf and Khan, Mohammad Emtiyaz}, volume = {249}, series = {Proceedings of Machine Learning Research}, month = {09 Oct}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v249/main/assets/yildirim24a/yildirim24a.pdf}, url = {https://proceedings.mlr.press/v249/yildirim24a.html}, abstract = {Class-Incremental Learning aims to update a deep classifier to learn new categories while maintaining or improving its accuracy on previously observed classes. Common methods to prevent forgetting previously learned classes include regularizing the neural network updates and storing exemplars in memory, which come with hyperparameters such as the learning rate, regularization strength, or the number of exemplars. However, these hyperparameters are usually only tuned at the start and then kept fixed throughout the learning sessions, ignoring the fact that newly encountered tasks may have varying levels of novelty or difficulty. This study investigates the necessity of hyperparameter ’adaptivity’ in Class-Incremental Learning: the ability to dynamically adjust hyperparameters such as the learning rate, regularization strength, and memory size according to the properties of the new task at hand. We propose AdaCL, a Bayesian Optimization-based approach to automatically and efficiently determine the optimal values for those parameters with each learning task. We show that adapting hyperpararmeters on each new task leads to improvement in accuracy, forgetting and memory. Code is available at https://github.com/ElifCerenGokYildirim/AdaCL.} }
Endnote
%0 Conference Paper %T AdaCL: Adaptive Continual Learning %A Elif Ceren Gok Yildirim %A Murat Onur Yildirim %A Mert Kilickaya %A Joaquin Vanschoren %B Proceedings of the 1st ContinualAI Unconference, 2023 %C Proceedings of Machine Learning Research %D 2024 %E Siddharth Swaroop %E Martin Mundt %E Rahaf Aljundi %E Mohammad Emtiyaz Khan %F pmlr-v249-yildirim24a %I PMLR %P 15--24 %U https://proceedings.mlr.press/v249/yildirim24a.html %V 249 %X Class-Incremental Learning aims to update a deep classifier to learn new categories while maintaining or improving its accuracy on previously observed classes. Common methods to prevent forgetting previously learned classes include regularizing the neural network updates and storing exemplars in memory, which come with hyperparameters such as the learning rate, regularization strength, or the number of exemplars. However, these hyperparameters are usually only tuned at the start and then kept fixed throughout the learning sessions, ignoring the fact that newly encountered tasks may have varying levels of novelty or difficulty. This study investigates the necessity of hyperparameter ’adaptivity’ in Class-Incremental Learning: the ability to dynamically adjust hyperparameters such as the learning rate, regularization strength, and memory size according to the properties of the new task at hand. We propose AdaCL, a Bayesian Optimization-based approach to automatically and efficiently determine the optimal values for those parameters with each learning task. We show that adapting hyperpararmeters on each new task leads to improvement in accuracy, forgetting and memory. Code is available at https://github.com/ElifCerenGokYildirim/AdaCL.
APA
Yildirim, E.C.G., Yildirim, M.O., Kilickaya, M. & Vanschoren, J.. (2024). AdaCL: Adaptive Continual Learning. Proceedings of the 1st ContinualAI Unconference, 2023, in Proceedings of Machine Learning Research 249:15-24 Available from https://proceedings.mlr.press/v249/yildirim24a.html.

Related Material