Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning

Arjun Prakash, Naicheng He, Kaicheng Guo, Saket Tiwari, Tyrone Serapio, Ruo Yu Tao, Amy Greenwald, George Konidaris
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:99578-99625, 2026.

Abstract

We investigate why deep neural networks suffer from loss of plasticity in continual learning, and thus fail to learn new tasks without reinitializing parameters. We show that this failure is preceded by Hessian spectral collapse at new-task initialization, where meaningful curvature directions vanish and gradient descent becomes ineffective. Analyzing a linearized ReLU network, we derive explicit $\epsilon$-rank conditions for successful training and prove that the loss-weighted Gram matrix is spectrally equivalent to the Generalized Gauss-Newton approximation, thereby relating NTK dynamics to Hessian curvature. Targeting spectral collapse directly, we then discuss the Kronecker factored approximation of the Hessian, which motivates two regularization enhancements: maintaining high effective feature rank and applying L2 penalties. Experiments on continual supervised and reinforcement learning tasks confirm that combining these two regularizers effectively preserves plasticity.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-prakash26b, title = {Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning}, author = {Prakash, Arjun and He, Naicheng and Guo, Kaicheng and Tiwari, Saket and Serapio, Tyrone and Tao, Ruo Yu and Greenwald, Amy and Konidaris, George}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {99578--99625}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/prakash26b/prakash26b.pdf}, url = {https://proceedings.mlr.press/v306/prakash26b.html}, abstract = {We investigate why deep neural networks suffer from loss of plasticity in continual learning, and thus fail to learn new tasks without reinitializing parameters. We show that this failure is preceded by Hessian spectral collapse at new-task initialization, where meaningful curvature directions vanish and gradient descent becomes ineffective. Analyzing a linearized ReLU network, we derive explicit $\epsilon$-rank conditions for successful training and prove that the loss-weighted Gram matrix is spectrally equivalent to the Generalized Gauss-Newton approximation, thereby relating NTK dynamics to Hessian curvature. Targeting spectral collapse directly, we then discuss the Kronecker factored approximation of the Hessian, which motivates two regularization enhancements: maintaining high effective feature rank and applying L2 penalties. Experiments on continual supervised and reinforcement learning tasks confirm that combining these two regularizers effectively preserves plasticity.} }
Endnote
%0 Conference Paper %T Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning %A Arjun Prakash %A Naicheng He %A Kaicheng Guo %A Saket Tiwari %A Tyrone Serapio %A Ruo Yu Tao %A Amy Greenwald %A George Konidaris %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-prakash26b %I PMLR %P 99578--99625 %U https://proceedings.mlr.press/v306/prakash26b.html %V 306 %X We investigate why deep neural networks suffer from loss of plasticity in continual learning, and thus fail to learn new tasks without reinitializing parameters. We show that this failure is preceded by Hessian spectral collapse at new-task initialization, where meaningful curvature directions vanish and gradient descent becomes ineffective. Analyzing a linearized ReLU network, we derive explicit $\epsilon$-rank conditions for successful training and prove that the loss-weighted Gram matrix is spectrally equivalent to the Generalized Gauss-Newton approximation, thereby relating NTK dynamics to Hessian curvature. Targeting spectral collapse directly, we then discuss the Kronecker factored approximation of the Hessian, which motivates two regularization enhancements: maintaining high effective feature rank and applying L2 penalties. Experiments on continual supervised and reinforcement learning tasks confirm that combining these two regularizers effectively preserves plasticity.
APA
Prakash, A., He, N., Guo, K., Tiwari, S., Serapio, T., Tao, R.Y., Greenwald, A. & Konidaris, G.. (2026). Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:99578-99625 Available from https://proceedings.mlr.press/v306/prakash26b.html.

Related Material