SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection

Zijie Peng, Enneng Yang, Yifei Cheng, Hongliang Yuan, Fei Ma, Xiaochun Cao, Li Shen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:98180-98202, 2026.

Abstract

Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-peng26n, title = {{SCNS}: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection}, author = {Peng, Zijie and Yang, Enneng and Cheng, Yifei and Yuan, Hongliang and Ma, Fei and Cao, Xiaochun and Shen, Li}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {98180--98202}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/peng26n/peng26n.pdf}, url = {https://proceedings.mlr.press/v306/peng26n.html}, abstract = {Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.} }
Endnote
%0 Conference Paper %T SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection %A Zijie Peng %A Enneng Yang %A Yifei Cheng %A Hongliang Yuan %A Fei Ma %A Xiaochun Cao %A Li Shen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-peng26n %I PMLR %P 98180--98202 %U https://proceedings.mlr.press/v306/peng26n.html %V 306 %X Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.
APA
Peng, Z., Yang, E., Cheng, Y., Yuan, H., Ma, F., Cao, X. & Shen, L.. (2026). SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:98180-98202 Available from https://proceedings.mlr.press/v306/peng26n.html.

Related Material