Convergence of Continual Learning in Homogeneous Deep Networks

Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry
Proceedings of Thirty Ninth Conference on Learning Theory, PMLR 336:5743-5784, 2026.

Abstract

We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v336-schliserman26a, title = {Convergence of Continual Learning in Homogeneous Deep Networks}, author = {Schliserman, Matan and Buzaglo, Gon and Evron, Itay and Soudry, Daniel}, booktitle = {Proceedings of Thirty Ninth Conference on Learning Theory}, pages = {5743--5784}, year = {2026}, editor = {Hanneke, Steve and Lattimore, Tor}, volume = {336}, series = {Proceedings of Machine Learning Research}, month = {29 Jun--03 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v336/main/assets/schliserman26a/schliserman26a.pdf}, url = {https://proceedings.mlr.press/v336/schliserman26a.html}, abstract = {We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.} }
Endnote
%0 Conference Paper %T Convergence of Continual Learning in Homogeneous Deep Networks %A Matan Schliserman %A Gon Buzaglo %A Itay Evron %A Daniel Soudry %B Proceedings of Thirty Ninth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2026 %E Steve Hanneke %E Tor Lattimore %F pmlr-v336-schliserman26a %I PMLR %P 5743--5784 %U https://proceedings.mlr.press/v336/schliserman26a.html %V 336 %X We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.
APA
Schliserman, M., Buzaglo, G., Evron, I. & Soudry, D.. (2026). Convergence of Continual Learning in Homogeneous Deep Networks. Proceedings of Thirty Ninth Conference on Learning Theory, in Proceedings of Machine Learning Research 336:5743-5784 Available from https://proceedings.mlr.press/v336/schliserman26a.html.

Related Material