Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks

Boqi Li, Youjun Wang, Weiwei Liu
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:36057-36095, 2025.

Abstract

Continual learning (CL) focuses on the ability of models to learn sequentially from a stream of tasks. A major challenge in CL is catastrophic forgetting (CF). CF is a phenomenon where the model experiences significant performance degradation on previously learned tasks after training on new tasks. Although CF is commonly observed in convolutional neural networks (CNNs), the theoretical understanding about CF within CNNs remains limited. To fill the gap, we present a theoretical analysis of CF in a two-layer CNN. By employing a multi-view data model, we analyze the learning dynamics of different features throughout CL and derive theoretical insights. The findings are supported by empirical results from both simulated and real-world datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-li25cm, title = {Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks}, author = {Li, Boqi and Wang, Youjun and Liu, Weiwei}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {36057--36095}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/li25cm/li25cm.pdf}, url = {https://proceedings.mlr.press/v267/li25cm.html}, abstract = {Continual learning (CL) focuses on the ability of models to learn sequentially from a stream of tasks. A major challenge in CL is catastrophic forgetting (CF). CF is a phenomenon where the model experiences significant performance degradation on previously learned tasks after training on new tasks. Although CF is commonly observed in convolutional neural networks (CNNs), the theoretical understanding about CF within CNNs remains limited. To fill the gap, we present a theoretical analysis of CF in a two-layer CNN. By employing a multi-view data model, we analyze the learning dynamics of different features throughout CL and derive theoretical insights. The findings are supported by empirical results from both simulated and real-world datasets.} }
Endnote
%0 Conference Paper %T Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks %A Boqi Li %A Youjun Wang %A Weiwei Liu %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-li25cm %I PMLR %P 36057--36095 %U https://proceedings.mlr.press/v267/li25cm.html %V 267 %X Continual learning (CL) focuses on the ability of models to learn sequentially from a stream of tasks. A major challenge in CL is catastrophic forgetting (CF). CF is a phenomenon where the model experiences significant performance degradation on previously learned tasks after training on new tasks. Although CF is commonly observed in convolutional neural networks (CNNs), the theoretical understanding about CF within CNNs remains limited. To fill the gap, we present a theoretical analysis of CF in a two-layer CNN. By employing a multi-view data model, we analyze the learning dynamics of different features throughout CL and derive theoretical insights. The findings are supported by empirical results from both simulated and real-world datasets.
APA
Li, B., Wang, Y. & Liu, W.. (2025). Towards Understanding Catastrophic Forgetting in Two-layer Convolutional Neural Networks. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:36057-36095 Available from https://proceedings.mlr.press/v267/li25cm.html.

Related Material