StrokeNUWA—Tokenizing Strokes for Vector Graphic Synthesis

Zecheng Tang, Chenfei Wu, Zekai Zhang, Minheng Ni, Shengming Yin, Yu Liu, Zhengyuan Yang, Lijuan Wang, Zicheng Liu, Juntao Li, Nan Duan
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:47830-47845, 2024.

Abstract

To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model’s ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation "stroke" tokens on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a $94\times$ speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-tang24h, title = {{S}troke{NUWA}—Tokenizing Strokes for Vector Graphic Synthesis}, author = {Tang, Zecheng and Wu, Chenfei and Zhang, Zekai and Ni, Minheng and Yin, Shengming and Liu, Yu and Yang, Zhengyuan and Wang, Lijuan and Liu, Zicheng and Li, Juntao and Duan, Nan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {47830--47845}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/tang24h/tang24h.pdf}, url = {https://proceedings.mlr.press/v235/tang24h.html}, abstract = {To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model’s ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation "stroke" tokens on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a $94\times$ speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.} }
Endnote
%0 Conference Paper %T StrokeNUWA—Tokenizing Strokes for Vector Graphic Synthesis %A Zecheng Tang %A Chenfei Wu %A Zekai Zhang %A Minheng Ni %A Shengming Yin %A Yu Liu %A Zhengyuan Yang %A Lijuan Wang %A Zicheng Liu %A Juntao Li %A Nan Duan %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-tang24h %I PMLR %P 47830--47845 %U https://proceedings.mlr.press/v235/tang24h.html %V 235 %X To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model’s ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation "stroke" tokens on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a $94\times$ speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.
APA
Tang, Z., Wu, C., Zhang, Z., Ni, M., Yin, S., Liu, Y., Yang, Z., Wang, L., Liu, Z., Li, J. & Duan, N.. (2024). StrokeNUWA—Tokenizing Strokes for Vector Graphic Synthesis. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:47830-47845 Available from https://proceedings.mlr.press/v235/tang24h.html.

Related Material