Continual Learning of Diffusion Models with Generative Distillation

Sergi Masip, Pau Rodriguez, Tinne Tuytelaars, Gido M van de Ven
Proceedings of The 3rd Conference on Lifelong Learning Agents, PMLR 274:431-456, 2025.

Abstract

Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus enabling the reuse of trained models for further learning. One potentially suitable continual learning approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach substantially improves the continual learning performance of generative replay with only a modest increase in the computational costs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v274-masip25a, title = {Continual Learning of Diffusion Models with Generative Distillation}, author = {Masip, Sergi and Rodriguez, Pau and Tuytelaars, Tinne and Ven, Gido M van de}, booktitle = {Proceedings of The 3rd Conference on Lifelong Learning Agents}, pages = {431--456}, year = {2025}, editor = {Lomonaco, Vincenzo and Melacci, Stefano and Tuytelaars, Tinne and Chandar, Sarath and Pascanu, Razvan}, volume = {274}, series = {Proceedings of Machine Learning Research}, month = {29 Jul--01 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v274/main/assets/masip25a/masip25a.pdf}, url = {https://proceedings.mlr.press/v274/masip25a.html}, abstract = {Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus enabling the reuse of trained models for further learning. One potentially suitable continual learning approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach substantially improves the continual learning performance of generative replay with only a modest increase in the computational costs.} }
Endnote
%0 Conference Paper %T Continual Learning of Diffusion Models with Generative Distillation %A Sergi Masip %A Pau Rodriguez %A Tinne Tuytelaars %A Gido M van de Ven %B Proceedings of The 3rd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2025 %E Vincenzo Lomonaco %E Stefano Melacci %E Tinne Tuytelaars %E Sarath Chandar %E Razvan Pascanu %F pmlr-v274-masip25a %I PMLR %P 431--456 %U https://proceedings.mlr.press/v274/masip25a.html %V 274 %X Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus enabling the reuse of trained models for further learning. One potentially suitable continual learning approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach substantially improves the continual learning performance of generative replay with only a modest increase in the computational costs.
APA
Masip, S., Rodriguez, P., Tuytelaars, T. & Ven, G.M.v.d.. (2025). Continual Learning of Diffusion Models with Generative Distillation. Proceedings of The 3rd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 274:431-456 Available from https://proceedings.mlr.press/v274/masip25a.html.

Related Material