Dual Optimal Transport for Multi-Concept Composition: Structure Alignment and Texture Injection in Diffusion Models

Hao Fu, Tianyu Su, Meng Liu, Chenfang Yang, Tian Gan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:32040-32052, 2026.

Abstract

Diffusion models have shown impressive capabilities in text-to-image synthesis. However, multi-concept personalized generation remains challenging, particularly in aligning multiple reference concepts while preserving fidelity. To address this, we propose a novel Sketch-to-Rendering framework that leverages $\textit{Dual Optimal Transport (OT)}$ for structure alignment and texture injection. Our approach consists of two key components: $\textit{Structure Sketching via Barycentric Soft-Transport}$, which ensures shape alignment by using mass-preserving OT for spatial consistency, and $\textit{Texture Rendering via Geometry-Guided Transport}$, which leverages low-frequency structure alignment to inject high-frequency texture details via OT-based residual transfer, thereby preserving texture fidelity without distorting structure. Extensive experiments demonstrate that our method significantly enhances both conceptual fidelity and visual quality. Ablation studies further validate the effectiveness of our optimal transport guidance and the decoupling of structure and texture during the generation process. Our code is available at https://github.com/fuhao7i/OTComp.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-fu26o, title = {Dual Optimal Transport for Multi-Concept Composition: Structure Alignment and Texture Injection in Diffusion Models}, author = {Fu, Hao and Su, Tianyu and Liu, Meng and Yang, Chenfang and Gan, Tian}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {32040--32052}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/fu26o/fu26o.pdf}, url = {https://proceedings.mlr.press/v306/fu26o.html}, abstract = {Diffusion models have shown impressive capabilities in text-to-image synthesis. However, multi-concept personalized generation remains challenging, particularly in aligning multiple reference concepts while preserving fidelity. To address this, we propose a novel Sketch-to-Rendering framework that leverages $\textit{Dual Optimal Transport (OT)}$ for structure alignment and texture injection. Our approach consists of two key components: $\textit{Structure Sketching via Barycentric Soft-Transport}$, which ensures shape alignment by using mass-preserving OT for spatial consistency, and $\textit{Texture Rendering via Geometry-Guided Transport}$, which leverages low-frequency structure alignment to inject high-frequency texture details via OT-based residual transfer, thereby preserving texture fidelity without distorting structure. Extensive experiments demonstrate that our method significantly enhances both conceptual fidelity and visual quality. Ablation studies further validate the effectiveness of our optimal transport guidance and the decoupling of structure and texture during the generation process. Our code is available at https://github.com/fuhao7i/OTComp.} }
Endnote
%0 Conference Paper %T Dual Optimal Transport for Multi-Concept Composition: Structure Alignment and Texture Injection in Diffusion Models %A Hao Fu %A Tianyu Su %A Meng Liu %A Chenfang Yang %A Tian Gan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-fu26o %I PMLR %P 32040--32052 %U https://proceedings.mlr.press/v306/fu26o.html %V 306 %X Diffusion models have shown impressive capabilities in text-to-image synthesis. However, multi-concept personalized generation remains challenging, particularly in aligning multiple reference concepts while preserving fidelity. To address this, we propose a novel Sketch-to-Rendering framework that leverages $\textit{Dual Optimal Transport (OT)}$ for structure alignment and texture injection. Our approach consists of two key components: $\textit{Structure Sketching via Barycentric Soft-Transport}$, which ensures shape alignment by using mass-preserving OT for spatial consistency, and $\textit{Texture Rendering via Geometry-Guided Transport}$, which leverages low-frequency structure alignment to inject high-frequency texture details via OT-based residual transfer, thereby preserving texture fidelity without distorting structure. Extensive experiments demonstrate that our method significantly enhances both conceptual fidelity and visual quality. Ablation studies further validate the effectiveness of our optimal transport guidance and the decoupling of structure and texture during the generation process. Our code is available at https://github.com/fuhao7i/OTComp.
APA
Fu, H., Su, T., Liu, M., Yang, C. & Gan, T.. (2026). Dual Optimal Transport for Multi-Concept Composition: Structure Alignment and Texture Injection in Diffusion Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:32040-32052 Available from https://proceedings.mlr.press/v306/fu26o.html.

Related Material