Forget-free Continual Learning with Winning Subnetworks

Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, Chang D. Yoo
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:10734-10750, 2022.

Abstract

Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-kang22b, title = {Forget-free Continual Learning with Winning Subnetworks}, author = {Kang, Haeyong and Mina, Rusty John Lloyd and Madjid, Sultan Rizky Hikmawan and Yoon, Jaehong and Hasegawa-Johnson, Mark and Hwang, Sung Ju and Yoo, Chang D.}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {10734--10750}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/kang22b/kang22b.pdf}, url = {https://proceedings.mlr.press/v162/kang22b.html}, abstract = {Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.} }
Endnote
%0 Conference Paper %T Forget-free Continual Learning with Winning Subnetworks %A Haeyong Kang %A Rusty John Lloyd Mina %A Sultan Rizky Hikmawan Madjid %A Jaehong Yoon %A Mark Hasegawa-Johnson %A Sung Ju Hwang %A Chang D. Yoo %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-kang22b %I PMLR %P 10734--10750 %U https://proceedings.mlr.press/v162/kang22b.html %V 162 %X Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.
APA
Kang, H., Mina, R.J.L., Madjid, S.R.H., Yoon, J., Hasegawa-Johnson, M., Hwang, S.J. & Yoo, C.D.. (2022). Forget-free Continual Learning with Winning Subnetworks. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:10734-10750 Available from https://proceedings.mlr.press/v162/kang22b.html.

Related Material