Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling

Zehao Dou, Minshuo Chen, Mengdi Wang, Zhuoran Yang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:11592-11612, 2024.

Abstract

Diffusion models have revolutionized various application domains, including computer vision and audio generation. Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved. In response, consistency models have been developed to merge multiple steps in the sampling process, thereby significantly boosting the speed of sample generation without compromising quality. This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem. Our analysis yields statistical estimation rates based on the Wasserstein distance for consistency models, matching those of vanilla diffusion models. Additionally, our results encompass the training of consistency models through both distillation and isolation methods, demystifying their underlying advantage.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-dou24a, title = {Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling}, author = {Dou, Zehao and Chen, Minshuo and Wang, Mengdi and Yang, Zhuoran}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {11592--11612}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/dou24a/dou24a.pdf}, url = {https://proceedings.mlr.press/v235/dou24a.html}, abstract = {Diffusion models have revolutionized various application domains, including computer vision and audio generation. Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved. In response, consistency models have been developed to merge multiple steps in the sampling process, thereby significantly boosting the speed of sample generation without compromising quality. This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem. Our analysis yields statistical estimation rates based on the Wasserstein distance for consistency models, matching those of vanilla diffusion models. Additionally, our results encompass the training of consistency models through both distillation and isolation methods, demystifying their underlying advantage.} }
Endnote
%0 Conference Paper %T Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling %A Zehao Dou %A Minshuo Chen %A Mengdi Wang %A Zhuoran Yang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-dou24a %I PMLR %P 11592--11612 %U https://proceedings.mlr.press/v235/dou24a.html %V 235 %X Diffusion models have revolutionized various application domains, including computer vision and audio generation. Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved. In response, consistency models have been developed to merge multiple steps in the sampling process, thereby significantly boosting the speed of sample generation without compromising quality. This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem. Our analysis yields statistical estimation rates based on the Wasserstein distance for consistency models, matching those of vanilla diffusion models. Additionally, our results encompass the training of consistency models through both distillation and isolation methods, demystifying their underlying advantage.
APA
Dou, Z., Chen, M., Wang, M. & Yang, Z.. (2024). Theory of Consistency Diffusion Models: Distribution Estimation Meets Fast Sampling. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:11592-11612 Available from https://proceedings.mlr.press/v235/dou24a.html.

Related Material