Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn

Hongyao Tang, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Glen Berseth
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:58883-58904, 2025.

Abstract

Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability induced by the data in each training batch. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-tang25g, title = {Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn}, author = {Tang, Hongyao and Obando-Ceron, Johan and Castro, Pablo Samuel and Courville, Aaron and Berseth, Glen}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {58883--58904}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/tang25g/tang25g.pdf}, url = {https://proceedings.mlr.press/v267/tang25g.html}, abstract = {Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability induced by the data in each training batch. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.} }
Endnote
%0 Conference Paper %T Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn %A Hongyao Tang %A Johan Obando-Ceron %A Pablo Samuel Castro %A Aaron Courville %A Glen Berseth %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-tang25g %I PMLR %P 58883--58904 %U https://proceedings.mlr.press/v267/tang25g.html %V 267 %X Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability induced by the data in each training batch. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.
APA
Tang, H., Obando-Ceron, J., Castro, P.S., Courville, A. & Berseth, G.. (2025). Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:58883-58904 Available from https://proceedings.mlr.press/v267/tang25g.html.

Related Material