Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale

Candi Zheng, Yuan Lan
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:61386-61412, 2024.

Abstract

Popular guidance for denoising diffusion probabilistic model (DDPM) linearly combines distinct conditional models together to provide enhanced control over samples. However, this approach overlooks nonlinear effects that become significant when guidance scale is large. To address this issue, we propose characteristic guidance, a guidance method that provides first-principle non-linear correction for classifier-free guidance. Such correction forces the guided DDPMs to respect the Fokker-Planck (FP) equation of diffusion process, in a way that is training-free and compatible with existing sampling methods. Experiments show that characteristic guidance enhances semantic characteristics of prompts and mitigate irregularities in image generation, proving effective in diverse applications ranging from simulating magnet phase transitions to latent space sampling.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zheng24f, title = {Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale}, author = {Zheng, Candi and Lan, Yuan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {61386--61412}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zheng24f/zheng24f.pdf}, url = {https://proceedings.mlr.press/v235/zheng24f.html}, abstract = {Popular guidance for denoising diffusion probabilistic model (DDPM) linearly combines distinct conditional models together to provide enhanced control over samples. However, this approach overlooks nonlinear effects that become significant when guidance scale is large. To address this issue, we propose characteristic guidance, a guidance method that provides first-principle non-linear correction for classifier-free guidance. Such correction forces the guided DDPMs to respect the Fokker-Planck (FP) equation of diffusion process, in a way that is training-free and compatible with existing sampling methods. Experiments show that characteristic guidance enhances semantic characteristics of prompts and mitigate irregularities in image generation, proving effective in diverse applications ranging from simulating magnet phase transitions to latent space sampling.} }
Endnote
%0 Conference Paper %T Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale %A Candi Zheng %A Yuan Lan %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zheng24f %I PMLR %P 61386--61412 %U https://proceedings.mlr.press/v235/zheng24f.html %V 235 %X Popular guidance for denoising diffusion probabilistic model (DDPM) linearly combines distinct conditional models together to provide enhanced control over samples. However, this approach overlooks nonlinear effects that become significant when guidance scale is large. To address this issue, we propose characteristic guidance, a guidance method that provides first-principle non-linear correction for classifier-free guidance. Such correction forces the guided DDPMs to respect the Fokker-Planck (FP) equation of diffusion process, in a way that is training-free and compatible with existing sampling methods. Experiments show that characteristic guidance enhances semantic characteristics of prompts and mitigate irregularities in image generation, proving effective in diverse applications ranging from simulating magnet phase transitions to latent space sampling.
APA
Zheng, C. & Lan, Y.. (2024). Characteristic Guidance: Non-linear Correction for Diffusion Model at Large Guidance Scale. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:61386-61412 Available from https://proceedings.mlr.press/v235/zheng24f.html.

Related Material