A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control

Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, Huazhe Xu
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:28921-28942, 2025.

Abstract

Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias—a tendency to overfit early experiences stored in the replay buffer—which limits an RL agent’s sample efficiency and generalizability. A common existing approach to mitigate this issue is periodically resetting the agent during training. Yet, even after multiple resets, RL agents could still be impacted by early experiences. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose Forget and Grow (FoG), a new deep RL algorithm with two mechanisms introduced. First, Experience Replay Decay (ER Decay)—"forgetting early experience”—which balances memory by gradually reducing the influence of early experiences. Second, Network Expansion—"growing neural capacity”—which enhances agents’ capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of FoG against SoTA existing deep RL algorithms, including BRO, SimBa and TD-MPC2.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-kang25c, title = {A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control}, author = {Kang, Zilin and Hu, Chenyuan and Luo, Yu and Yuan, Zhecheng and Zheng, Ruijie and Xu, Huazhe}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {28921--28942}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/kang25c/kang25c.pdf}, url = {https://proceedings.mlr.press/v267/kang25c.html}, abstract = {Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias—a tendency to overfit early experiences stored in the replay buffer—which limits an RL agent’s sample efficiency and generalizability. A common existing approach to mitigate this issue is periodically resetting the agent during training. Yet, even after multiple resets, RL agents could still be impacted by early experiences. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose Forget and Grow (FoG), a new deep RL algorithm with two mechanisms introduced. First, Experience Replay Decay (ER Decay)—"forgetting early experience”—which balances memory by gradually reducing the influence of early experiences. Second, Network Expansion—"growing neural capacity”—which enhances agents’ capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of FoG against SoTA existing deep RL algorithms, including BRO, SimBa and TD-MPC2.} }
Endnote
%0 Conference Paper %T A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control %A Zilin Kang %A Chenyuan Hu %A Yu Luo %A Zhecheng Yuan %A Ruijie Zheng %A Huazhe Xu %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-kang25c %I PMLR %P 28921--28942 %U https://proceedings.mlr.press/v267/kang25c.html %V 267 %X Deep reinforcement learning for continuous control has recently achieved impressive progress. However, existing methods often suffer from primacy bias—a tendency to overfit early experiences stored in the replay buffer—which limits an RL agent’s sample efficiency and generalizability. A common existing approach to mitigate this issue is periodically resetting the agent during training. Yet, even after multiple resets, RL agents could still be impacted by early experiences. In contrast, humans are less susceptible to such bias, partly due to infantile amnesia, where the formation of new neurons disrupts early memory traces, leading to the forgetting of initial experiences. Inspired by this dual processes of forgetting and growing in neuroscience, in this paper, we propose Forget and Grow (FoG), a new deep RL algorithm with two mechanisms introduced. First, Experience Replay Decay (ER Decay)—"forgetting early experience”—which balances memory by gradually reducing the influence of early experiences. Second, Network Expansion—"growing neural capacity”—which enhances agents’ capability to exploit the patterns of existing data by dynamically adding new parameters during training. Empirical results on four major continuous control benchmarks with more than 40 tasks demonstrate the superior performance of FoG against SoTA existing deep RL algorithms, including BRO, SimBa and TD-MPC2.
APA
Kang, Z., Hu, C., Luo, Y., Yuan, Z., Zheng, R. & Xu, H.. (2025). A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:28921-28942 Available from https://proceedings.mlr.press/v267/kang25c.html.

Related Material