Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models

Xiaoling Gu, Xuelong Li, Shengqi Wu, Yongkang Wong, Zizhao Wu, Huan Li, Zhou Yu, Mohan Kankanhalli
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:37047-37072, 2026.

Abstract

Despite remarkable progress in text-to-image diffusion models, accurately generating the specified number of objects remains a persistent challenge. We identify the initial noise as a primary determinant of spatial layout formation, with early-stage cross-attention serving as the key mechanism that mediates the propagation of noise-induced structures throughout the denoising process. We characterize this phenomenon as Noise-Induced Layout Prior. Leveraging this insight, we propose a novel training-free framework for object counting in diffusion models. Our approach consists of two key components: (1) a Count-Aware Noise Adjustment Strategy, which explicitly manipulates the initial latent noise to align layout formation with the target object count, and (2) an Attention-Guided Layout Consistency Strategy, which performs test-time optimization on early-stage cross-attention to further stabilize layout formation during denoising. Extensive experiments on both single-category and multi-category benchmarks demonstrate that our method consistently outperforms strong diffusion baselines and state-of-the-art object count control methods in terms of counting accuracy and image quality. Code Release: https://github.com/lxlong1201/Mitigate_Noise_Prior.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gu26f, title = {Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models}, author = {Gu, Xiaoling and Li, Xuelong and Wu, Shengqi and Wong, Yongkang and Wu, Zizhao and Li, Huan and Yu, Zhou and Kankanhalli, Mohan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {37047--37072}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gu26f/gu26f.pdf}, url = {https://proceedings.mlr.press/v306/gu26f.html}, abstract = {Despite remarkable progress in text-to-image diffusion models, accurately generating the specified number of objects remains a persistent challenge. We identify the initial noise as a primary determinant of spatial layout formation, with early-stage cross-attention serving as the key mechanism that mediates the propagation of noise-induced structures throughout the denoising process. We characterize this phenomenon as Noise-Induced Layout Prior. Leveraging this insight, we propose a novel training-free framework for object counting in diffusion models. Our approach consists of two key components: (1) a Count-Aware Noise Adjustment Strategy, which explicitly manipulates the initial latent noise to align layout formation with the target object count, and (2) an Attention-Guided Layout Consistency Strategy, which performs test-time optimization on early-stage cross-attention to further stabilize layout formation during denoising. Extensive experiments on both single-category and multi-category benchmarks demonstrate that our method consistently outperforms strong diffusion baselines and state-of-the-art object count control methods in terms of counting accuracy and image quality. Code Release: https://github.com/lxlong1201/Mitigate_Noise_Prior.} }
Endnote
%0 Conference Paper %T Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models %A Xiaoling Gu %A Xuelong Li %A Shengqi Wu %A Yongkang Wong %A Zizhao Wu %A Huan Li %A Zhou Yu %A Mohan Kankanhalli %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gu26f %I PMLR %P 37047--37072 %U https://proceedings.mlr.press/v306/gu26f.html %V 306 %X Despite remarkable progress in text-to-image diffusion models, accurately generating the specified number of objects remains a persistent challenge. We identify the initial noise as a primary determinant of spatial layout formation, with early-stage cross-attention serving as the key mechanism that mediates the propagation of noise-induced structures throughout the denoising process. We characterize this phenomenon as Noise-Induced Layout Prior. Leveraging this insight, we propose a novel training-free framework for object counting in diffusion models. Our approach consists of two key components: (1) a Count-Aware Noise Adjustment Strategy, which explicitly manipulates the initial latent noise to align layout formation with the target object count, and (2) an Attention-Guided Layout Consistency Strategy, which performs test-time optimization on early-stage cross-attention to further stabilize layout formation during denoising. Extensive experiments on both single-category and multi-category benchmarks demonstrate that our method consistently outperforms strong diffusion baselines and state-of-the-art object count control methods in terms of counting accuracy and image quality. Code Release: https://github.com/lxlong1201/Mitigate_Noise_Prior.
APA
Gu, X., Li, X., Wu, S., Wong, Y., Wu, Z., Li, H., Yu, Z. & Kankanhalli, M.. (2026). Mitigating Noise-Induced Layout Priors for Object Counting in Diffusion Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:37047-37072 Available from https://proceedings.mlr.press/v306/gu26f.html.

Related Material