Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss

Sangyeon Park, Isaac Han, Seungwon Oh, Kyungjoong Kim
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:47991-48026, 2025.

Abstract

Plasticity loss, a critical challenge in neural network training, limits a model’s ability to adapt to new tasks or shifts in data distribution. While widely used techniques like L2 regularization and Layer Normalization have proven effective in mitigating this issue, Dropout remains notably ineffective. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-park25b, title = {Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss}, author = {Park, Sangyeon and Han, Isaac and Oh, Seungwon and Kim, Kyungjoong}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {47991--48026}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/park25b/park25b.pdf}, url = {https://proceedings.mlr.press/v267/park25b.html}, abstract = {Plasticity loss, a critical challenge in neural network training, limits a model’s ability to adapt to new tasks or shifts in data distribution. While widely used techniques like L2 regularization and Layer Normalization have proven effective in mitigating this issue, Dropout remains notably ineffective. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.} }
Endnote
%0 Conference Paper %T Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss %A Sangyeon Park %A Isaac Han %A Seungwon Oh %A Kyungjoong Kim %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-park25b %I PMLR %P 47991--48026 %U https://proceedings.mlr.press/v267/park25b.html %V 267 %X Plasticity loss, a critical challenge in neural network training, limits a model’s ability to adapt to new tasks or shifts in data distribution. While widely used techniques like L2 regularization and Layer Normalization have proven effective in mitigating this issue, Dropout remains notably ineffective. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.
APA
Park, S., Han, I., Oh, S. & Kim, K.. (2025). Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:47991-48026 Available from https://proceedings.mlr.press/v267/park25b.html.

Related Material