Continual Learning in Linear Classification on Separable Data

Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, Daniel Soudry
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:9440-9484, 2023.

Abstract

We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-evron23a, title = {Continual Learning in Linear Classification on Separable Data}, author = {Evron, Itay and Moroshko, Edward and Buzaglo, Gon and Khriesh, Maroun and Marjieh, Badea and Srebro, Nathan and Soudry, Daniel}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {9440--9484}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/evron23a/evron23a.pdf}, url = {https://proceedings.mlr.press/v202/evron23a.html}, abstract = {We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.} }
Endnote
%0 Conference Paper %T Continual Learning in Linear Classification on Separable Data %A Itay Evron %A Edward Moroshko %A Gon Buzaglo %A Maroun Khriesh %A Badea Marjieh %A Nathan Srebro %A Daniel Soudry %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-evron23a %I PMLR %P 9440--9484 %U https://proceedings.mlr.press/v202/evron23a.html %V 202 %X We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to solving a sequential max-margin problem, corresponding to a special case of the Projection Onto Convex Sets (POCS) framework. We then develop upper bounds on the forgetting and other quantities of interest under various settings with recurring tasks, including cyclic and random orderings of tasks. We discuss several practical implications to popular training practices like regularization scheduling and weighting. We point out several theoretical differences between our continual classification setting and a recently studied continual regression setting.
APA
Evron, I., Moroshko, E., Buzaglo, G., Khriesh, M., Marjieh, B., Srebro, N. & Soudry, D.. (2023). Continual Learning in Linear Classification on Separable Data. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:9440-9484 Available from https://proceedings.mlr.press/v202/evron23a.html.

Related Material