Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning

Hongxi Li, Tong Wang, Wu Chengjing, Tianbao Liu, Jiangtao Yao, Xiaochao Qu, Xinxiao Wu, Luoqi Liu, Ting Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:70603-70622, 2026.

Abstract

Scene text editing aims to modify text in a target region of an image while preserving its background style and texture. Existing methods rely solely on image background information while neglecting the visual details of target regions, which discards stylistic features in the original text and essentially degrades the task to text rendering. Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text. To address these issues, this paper proposes a self-prompting scene text editing method, which constructs style and glyph prompts directly from the original image without additional style or glyph encoders. We employ a two-stage training strategy, where the diffusion transformer is first trained on large-scale self-supervised datasets and subsequently refined with a small set of paired images. By leveraging the in-context learning capability of FLUX-Fill, it achieves open-vocabulary and style-consistent text editing. Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-li26gj, title = {Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning}, author = {Li, Hongxi and Wang, Tong and Chengjing, Wu and Liu, Tianbao and Yao, Jiangtao and Qu, Xiaochao and Wu, Xinxiao and Liu, Luoqi and Liu, Ting}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {70603--70622}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26gj/li26gj.pdf}, url = {https://proceedings.mlr.press/v306/li26gj.html}, abstract = {Scene text editing aims to modify text in a target region of an image while preserving its background style and texture. Existing methods rely solely on image background information while neglecting the visual details of target regions, which discards stylistic features in the original text and essentially degrades the task to text rendering. Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text. To address these issues, this paper proposes a self-prompting scene text editing method, which constructs style and glyph prompts directly from the original image without additional style or glyph encoders. We employ a two-stage training strategy, where the diffusion transformer is first trained on large-scale self-supervised datasets and subsequently refined with a small set of paired images. By leveraging the in-context learning capability of FLUX-Fill, it achieves open-vocabulary and style-consistent text editing. Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency.} }
Endnote
%0 Conference Paper %T Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning %A Hongxi Li %A Tong Wang %A Wu Chengjing %A Tianbao Liu %A Jiangtao Yao %A Xiaochao Qu %A Xinxiao Wu %A Luoqi Liu %A Ting Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26gj %I PMLR %P 70603--70622 %U https://proceedings.mlr.press/v306/li26gj.html %V 306 %X Scene text editing aims to modify text in a target region of an image while preserving its background style and texture. Existing methods rely solely on image background information while neglecting the visual details of target regions, which discards stylistic features in the original text and essentially degrades the task to text rendering. Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text. To address these issues, this paper proposes a self-prompting scene text editing method, which constructs style and glyph prompts directly from the original image without additional style or glyph encoders. We employ a two-stage training strategy, where the diffusion transformer is first trained on large-scale self-supervised datasets and subsequently refined with a small set of paired images. By leveraging the in-context learning capability of FLUX-Fill, it achieves open-vocabulary and style-consistent text editing. Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency.
APA
Li, H., Wang, T., Chengjing, W., Liu, T., Yao, J., Qu, X., Wu, X., Liu, L. & Liu, T.. (2026). Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:70603-70622 Available from https://proceedings.mlr.press/v306/li26gj.html.

Related Material