ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

Jinho Chang, Changsun Lee, Hyungjin Chung, Jong Chul Ye
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:12738-12757, 2026.

Abstract

As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term as a Negative Prompting (NP) to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to achieve guidance toward the given condition using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a similar guiding effect to traditional CFG for positive conditions while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively injects or removes the given concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chang26c, title = {{C}ontrastive{CFG}: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts}, author = {Chang, Jinho and Lee, Changsun and Chung, Hyungjin and Ye, Jong Chul}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {12738--12757}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chang26c/chang26c.pdf}, url = {https://proceedings.mlr.press/v306/chang26c.html}, abstract = {As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term as a Negative Prompting (NP) to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to achieve guidance toward the given condition using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a similar guiding effect to traditional CFG for positive conditions while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively injects or removes the given concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.} }
Endnote
%0 Conference Paper %T ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts %A Jinho Chang %A Changsun Lee %A Hyungjin Chung %A Jong Chul Ye %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chang26c %I PMLR %P 12738--12757 %U https://proceedings.mlr.press/v306/chang26c.html %V 306 %X As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term as a Negative Prompting (NP) to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to achieve guidance toward the given condition using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a similar guiding effect to traditional CFG for positive conditions while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively injects or removes the given concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.
APA
Chang, J., Lee, C., Chung, H. & Ye, J.C.. (2026). ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:12738-12757 Available from https://proceedings.mlr.press/v306/chang26c.html.

Related Material