Addressing Catastrophic Forgetting in Few-Shot Problems

Pauching Yap, Hippolyt Ritter, David Barber
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11909-11919, 2021.

Abstract

Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has been done to overcome catastrophic forgetting in few-shot classification problems. We demonstrate that the popular gradient-based model-agnostic meta-learning algorithm (MAML) indeed suffers from catastrophic forgetting and introduce a Bayesian online meta-learning framework that tackles this problem. Our framework utilises Bayesian online learning and meta-learning along with Laplace approximation and variational inference to overcome catastrophic forgetting in few-shot classification problems. The experimental evaluations demonstrate that our framework can effectively achieve this goal in comparison with various baselines. As an additional utility, we also demonstrate empirically that our framework is capable of meta-learning on sequentially arriving few-shot tasks from a stationary task distribution.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-yap21a, title = {Addressing Catastrophic Forgetting in Few-Shot Problems}, author = {Yap, Pauching and Ritter, Hippolyt and Barber, David}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {11909--11919}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/yap21a/yap21a.pdf}, url = {https://proceedings.mlr.press/v139/yap21a.html}, abstract = {Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has been done to overcome catastrophic forgetting in few-shot classification problems. We demonstrate that the popular gradient-based model-agnostic meta-learning algorithm (MAML) indeed suffers from catastrophic forgetting and introduce a Bayesian online meta-learning framework that tackles this problem. Our framework utilises Bayesian online learning and meta-learning along with Laplace approximation and variational inference to overcome catastrophic forgetting in few-shot classification problems. The experimental evaluations demonstrate that our framework can effectively achieve this goal in comparison with various baselines. As an additional utility, we also demonstrate empirically that our framework is capable of meta-learning on sequentially arriving few-shot tasks from a stationary task distribution.} }
Endnote
%0 Conference Paper %T Addressing Catastrophic Forgetting in Few-Shot Problems %A Pauching Yap %A Hippolyt Ritter %A David Barber %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-yap21a %I PMLR %P 11909--11919 %U https://proceedings.mlr.press/v139/yap21a.html %V 139 %X Neural networks are known to suffer from catastrophic forgetting when trained on sequential datasets. While there have been numerous attempts to solve this problem in large-scale supervised classification, little has been done to overcome catastrophic forgetting in few-shot classification problems. We demonstrate that the popular gradient-based model-agnostic meta-learning algorithm (MAML) indeed suffers from catastrophic forgetting and introduce a Bayesian online meta-learning framework that tackles this problem. Our framework utilises Bayesian online learning and meta-learning along with Laplace approximation and variational inference to overcome catastrophic forgetting in few-shot classification problems. The experimental evaluations demonstrate that our framework can effectively achieve this goal in comparison with various baselines. As an additional utility, we also demonstrate empirically that our framework is capable of meta-learning on sequentially arriving few-shot tasks from a stationary task distribution.
APA
Yap, P., Ritter, H. & Barber, D.. (2021). Addressing Catastrophic Forgetting in Few-Shot Problems. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:11909-11919 Available from https://proceedings.mlr.press/v139/yap21a.html.

Related Material