AnyCanvas: Potential Field Guidance for Training-Free Spatial Control in Text-to-Image Diffusion

Tianyi Xie, Zhiyuan Yu, Kaihong Huang, Beilun Wang, Zhaoyang Wang, Dian Shen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:139803-139816, 2026.

Abstract

Diffusion-based text-to-image (T2I) models have demonstrated remarkable advancements in generating high-quality images. However, while real-world applications like product packaging and logo design necessitate synthesis within irregular geometries, existing methods struggle to handle such constraints. Therefore, generating complete pictures that conform to arbitrary-shaped canvas constraints while maintaining semantic integrity remains a significant challenge. To address this, we introduce AnyCanvas, a training-free framework that leverages a Mask-to-Potential Field paradigm to convert binary masks into a differentiable potential field, which guides content to naturally converge within target regions. Extensive experiments demonstrate that AnyCanvas achieves 4.23% higher spatial adherence to user-specified constraints while maintaining 99.45% of the semantic fidelity measured by CLIP score, leading to a superior harmonic mean of spatial and semantic metrics. AnyCanvas also exhibits robust generalizability across different model backbones and versatile spatial control objectives.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-xie26ab, title = {{A}ny{C}anvas: Potential Field Guidance for Training-Free Spatial Control in Text-to-Image Diffusion}, author = {Xie, Tianyi and Yu, Zhiyuan and Huang, Kaihong and Wang, Beilun and Wang, Zhaoyang and Shen, Dian}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {139803--139816}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xie26ab/xie26ab.pdf}, url = {https://proceedings.mlr.press/v306/xie26ab.html}, abstract = {Diffusion-based text-to-image (T2I) models have demonstrated remarkable advancements in generating high-quality images. However, while real-world applications like product packaging and logo design necessitate synthesis within irregular geometries, existing methods struggle to handle such constraints. Therefore, generating complete pictures that conform to arbitrary-shaped canvas constraints while maintaining semantic integrity remains a significant challenge. To address this, we introduce AnyCanvas, a training-free framework that leverages a Mask-to-Potential Field paradigm to convert binary masks into a differentiable potential field, which guides content to naturally converge within target regions. Extensive experiments demonstrate that AnyCanvas achieves 4.23% higher spatial adherence to user-specified constraints while maintaining 99.45% of the semantic fidelity measured by CLIP score, leading to a superior harmonic mean of spatial and semantic metrics. AnyCanvas also exhibits robust generalizability across different model backbones and versatile spatial control objectives.} }
Endnote
%0 Conference Paper %T AnyCanvas: Potential Field Guidance for Training-Free Spatial Control in Text-to-Image Diffusion %A Tianyi Xie %A Zhiyuan Yu %A Kaihong Huang %A Beilun Wang %A Zhaoyang Wang %A Dian Shen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xie26ab %I PMLR %P 139803--139816 %U https://proceedings.mlr.press/v306/xie26ab.html %V 306 %X Diffusion-based text-to-image (T2I) models have demonstrated remarkable advancements in generating high-quality images. However, while real-world applications like product packaging and logo design necessitate synthesis within irregular geometries, existing methods struggle to handle such constraints. Therefore, generating complete pictures that conform to arbitrary-shaped canvas constraints while maintaining semantic integrity remains a significant challenge. To address this, we introduce AnyCanvas, a training-free framework that leverages a Mask-to-Potential Field paradigm to convert binary masks into a differentiable potential field, which guides content to naturally converge within target regions. Extensive experiments demonstrate that AnyCanvas achieves 4.23% higher spatial adherence to user-specified constraints while maintaining 99.45% of the semantic fidelity measured by CLIP score, leading to a superior harmonic mean of spatial and semantic metrics. AnyCanvas also exhibits robust generalizability across different model backbones and versatile spatial control objectives.
APA
Xie, T., Yu, Z., Huang, K., Wang, B., Wang, Z. & Shen, D.. (2026). AnyCanvas: Potential Field Guidance for Training-Free Spatial Control in Text-to-Image Diffusion. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:139803-139816 Available from https://proceedings.mlr.press/v306/xie26ab.html.

Related Material