Full Network Capacity Framework for Sample-Efficient Deep Reinforcement Learning

Wentao Yang, Xinyue Liu, Yunlong Gao, Wenxin Liang, Linlin Zong, Guanglu Wang, Xianchao Zhang
Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, PMLR 286:4701-4714, 2025.

Abstract

In deep reinforcement learning (DRL), the presence of dormant neurons leads to a significant reduction in network capacity, which results in sub-optimal performance and limited sample efficiency. Existing training techniques, especially those relying on periodic resetting (PR), exacerbate this issue. We propose the Full Network Capacity (FNC) framework based on PR, which consists of two novel modules: Dormant Neuron Reactivation (DNR) and Stable Policy Update (SPU). DNR continuously reactivates dormant neurons, thereby enhancing network capacity. SPU mitigates perturbation from DNR and PR and stabilizes the Q-values for the actor, ensuring smooth training and reliable policy updates. Our experimental evaluations on the Atari 100K and DMControl 100K benchmarks demonstrate the remarkable sample efficiency of FNC. On Atari 100K, FNC achieves a superhuman IQM HNS of 107.3%, outperforming the previous state-of-the-art method BBF by 13.3%. On DMControl 100K, FNC excels in 5 out of 6 tasks in terms of episodic return and attains the highest median and mean aggregated scores. FNC not only maximizes network capacity but also provides a practical solution for real-world applications where data collection is costly and time-consuming. Our implementation is publicly accessible at \url{https://github.com/tlyy/FNC}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v286-yang25a, title = {Full Network Capacity Framework for Sample-Efficient Deep Reinforcement Learning}, author = {Yang, Wentao and Liu, Xinyue and Gao, Yunlong and Liang, Wenxin and Zong, Linlin and Wang, Guanglu and Zhang, Xianchao}, booktitle = {Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence}, pages = {4701--4714}, year = {2025}, editor = {Chiappa, Silvia and Magliacane, Sara}, volume = {286}, series = {Proceedings of Machine Learning Research}, month = {21--25 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v286/main/assets/yang25a/yang25a.pdf}, url = {https://proceedings.mlr.press/v286/yang25a.html}, abstract = {In deep reinforcement learning (DRL), the presence of dormant neurons leads to a significant reduction in network capacity, which results in sub-optimal performance and limited sample efficiency. Existing training techniques, especially those relying on periodic resetting (PR), exacerbate this issue. We propose the Full Network Capacity (FNC) framework based on PR, which consists of two novel modules: Dormant Neuron Reactivation (DNR) and Stable Policy Update (SPU). DNR continuously reactivates dormant neurons, thereby enhancing network capacity. SPU mitigates perturbation from DNR and PR and stabilizes the Q-values for the actor, ensuring smooth training and reliable policy updates. Our experimental evaluations on the Atari 100K and DMControl 100K benchmarks demonstrate the remarkable sample efficiency of FNC. On Atari 100K, FNC achieves a superhuman IQM HNS of 107.3%, outperforming the previous state-of-the-art method BBF by 13.3%. On DMControl 100K, FNC excels in 5 out of 6 tasks in terms of episodic return and attains the highest median and mean aggregated scores. FNC not only maximizes network capacity but also provides a practical solution for real-world applications where data collection is costly and time-consuming. Our implementation is publicly accessible at \url{https://github.com/tlyy/FNC}.} }
Endnote
%0 Conference Paper %T Full Network Capacity Framework for Sample-Efficient Deep Reinforcement Learning %A Wentao Yang %A Xinyue Liu %A Yunlong Gao %A Wenxin Liang %A Linlin Zong %A Guanglu Wang %A Xianchao Zhang %B Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2025 %E Silvia Chiappa %E Sara Magliacane %F pmlr-v286-yang25a %I PMLR %P 4701--4714 %U https://proceedings.mlr.press/v286/yang25a.html %V 286 %X In deep reinforcement learning (DRL), the presence of dormant neurons leads to a significant reduction in network capacity, which results in sub-optimal performance and limited sample efficiency. Existing training techniques, especially those relying on periodic resetting (PR), exacerbate this issue. We propose the Full Network Capacity (FNC) framework based on PR, which consists of two novel modules: Dormant Neuron Reactivation (DNR) and Stable Policy Update (SPU). DNR continuously reactivates dormant neurons, thereby enhancing network capacity. SPU mitigates perturbation from DNR and PR and stabilizes the Q-values for the actor, ensuring smooth training and reliable policy updates. Our experimental evaluations on the Atari 100K and DMControl 100K benchmarks demonstrate the remarkable sample efficiency of FNC. On Atari 100K, FNC achieves a superhuman IQM HNS of 107.3%, outperforming the previous state-of-the-art method BBF by 13.3%. On DMControl 100K, FNC excels in 5 out of 6 tasks in terms of episodic return and attains the highest median and mean aggregated scores. FNC not only maximizes network capacity but also provides a practical solution for real-world applications where data collection is costly and time-consuming. Our implementation is publicly accessible at \url{https://github.com/tlyy/FNC}.
APA
Yang, W., Liu, X., Gao, Y., Liang, W., Zong, L., Wang, G. & Zhang, X.. (2025). Full Network Capacity Framework for Sample-Efficient Deep Reinforcement Learning. Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 286:4701-4714 Available from https://proceedings.mlr.press/v286/yang25a.html.

Related Material