Concept-Guided Tokenization: Closing the Gap Between Reconstruction and Generation

Yunqiao Yang, Haokun Lin, Guanzhong Wu, Ying Wei
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:145424-145443, 2026.

Abstract

Recent advances in image generation have been largely driven by image tokenization, which compresses raw pixels into compact latent representations. While existing tokenizers excel at preserving low-level visual details through reconstruction-based training, they often lack explicit semantic guidance, which limits their ability to capture semantically structured representations and thus hinders their performance on downstream tasks like image generation. To overcome this limitation, we propose a novel tokenization framework that incorporates high-level semantics through two key innovations: (1) a text-integrated encoder that jointly processes images and textual descriptions to produce semantically enriched latent representations, and (2) a concept-guided training objective that leverages sparse autoencoders to decompose pre-trained vision-language model features to a semantic concept space, employing sparse and disentangled concept indices for guidance. Our approach achieves strong alignment with semantic concepts, maintaining high reconstruction fidelity with an rFID of 1.39 on ImageNet, while achieving a gFID of 2.65 on the class-conditional image generation task and 10.73 on the text-to-image generation task. By infusing high-level semantic structures into low-level visual fidelity, our method bridges the reconstruction-generation divide and drives generative modeling as a powerful foundation. The code is available at https://github.com/hustyyq/ConceptTok

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-yang26as, title = {Concept-Guided Tokenization: Closing the Gap Between Reconstruction and Generation}, author = {Yang, Yunqiao and Lin, Haokun and Wu, Guanzhong and Wei, Ying}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {145424--145443}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yang26as/yang26as.pdf}, url = {https://proceedings.mlr.press/v306/yang26as.html}, abstract = {Recent advances in image generation have been largely driven by image tokenization, which compresses raw pixels into compact latent representations. While existing tokenizers excel at preserving low-level visual details through reconstruction-based training, they often lack explicit semantic guidance, which limits their ability to capture semantically structured representations and thus hinders their performance on downstream tasks like image generation. To overcome this limitation, we propose a novel tokenization framework that incorporates high-level semantics through two key innovations: (1) a text-integrated encoder that jointly processes images and textual descriptions to produce semantically enriched latent representations, and (2) a concept-guided training objective that leverages sparse autoencoders to decompose pre-trained vision-language model features to a semantic concept space, employing sparse and disentangled concept indices for guidance. Our approach achieves strong alignment with semantic concepts, maintaining high reconstruction fidelity with an rFID of 1.39 on ImageNet, while achieving a gFID of 2.65 on the class-conditional image generation task and 10.73 on the text-to-image generation task. By infusing high-level semantic structures into low-level visual fidelity, our method bridges the reconstruction-generation divide and drives generative modeling as a powerful foundation. The code is available at https://github.com/hustyyq/ConceptTok} }
Endnote
%0 Conference Paper %T Concept-Guided Tokenization: Closing the Gap Between Reconstruction and Generation %A Yunqiao Yang %A Haokun Lin %A Guanzhong Wu %A Ying Wei %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yang26as %I PMLR %P 145424--145443 %U https://proceedings.mlr.press/v306/yang26as.html %V 306 %X Recent advances in image generation have been largely driven by image tokenization, which compresses raw pixels into compact latent representations. While existing tokenizers excel at preserving low-level visual details through reconstruction-based training, they often lack explicit semantic guidance, which limits their ability to capture semantically structured representations and thus hinders their performance on downstream tasks like image generation. To overcome this limitation, we propose a novel tokenization framework that incorporates high-level semantics through two key innovations: (1) a text-integrated encoder that jointly processes images and textual descriptions to produce semantically enriched latent representations, and (2) a concept-guided training objective that leverages sparse autoencoders to decompose pre-trained vision-language model features to a semantic concept space, employing sparse and disentangled concept indices for guidance. Our approach achieves strong alignment with semantic concepts, maintaining high reconstruction fidelity with an rFID of 1.39 on ImageNet, while achieving a gFID of 2.65 on the class-conditional image generation task and 10.73 on the text-to-image generation task. By infusing high-level semantic structures into low-level visual fidelity, our method bridges the reconstruction-generation divide and drives generative modeling as a powerful foundation. The code is available at https://github.com/hustyyq/ConceptTok
APA
Yang, Y., Lin, H., Wu, G. & Wei, Y.. (2026). Concept-Guided Tokenization: Closing the Gap Between Reconstruction and Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:145424-145443 Available from https://proceedings.mlr.press/v306/yang26as.html.

Related Material