Scaling Transformers for End-to-End Discrete Audio Tokenization

Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruixiao Li, Qinyuan Cheng, Shimin Li, Xipeng Qiu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:35903-35924, 2026.

Abstract

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. Based on this perspective, we propose $\textbf{TAC}$, a Transformer-based audio tokenizer that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction of general audio. We show that a simple, fully end-to-end learned tokenizer built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, the proposed tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging TAC’s discrete tokens, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, TAC enables competitive ASR performance without auxiliary encoders. Our findings position TAC as a unified, scalable interface for the next generation of native audio foundation models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gong26c, title = {Scaling Transformers for End-to-End Discrete Audio Tokenization}, author = {Gong, Yitian and Chen, Kuangwei and Fei, Zhaoye and Yang, Xiaogui and Chen, Ke and Wang, Yang and Huang, Kexin and Chen, Mingshu and Li, Ruixiao and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {35903--35924}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gong26c/gong26c.pdf}, url = {https://proceedings.mlr.press/v306/gong26c.html}, abstract = {Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. Based on this perspective, we propose $\textbf{TAC}$, a Transformer-based audio tokenizer that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction of general audio. We show that a simple, fully end-to-end learned tokenizer built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, the proposed tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging TAC’s discrete tokens, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, TAC enables competitive ASR performance without auxiliary encoders. Our findings position TAC as a unified, scalable interface for the next generation of native audio foundation models.} }
Endnote
%0 Conference Paper %T Scaling Transformers for End-to-End Discrete Audio Tokenization %A Yitian Gong %A Kuangwei Chen %A Zhaoye Fei %A Xiaogui Yang %A Ke Chen %A Yang Wang %A Kexin Huang %A Mingshu Chen %A Ruixiao Li %A Qinyuan Cheng %A Shimin Li %A Xipeng Qiu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gong26c %I PMLR %P 35903--35924 %U https://proceedings.mlr.press/v306/gong26c.html %V 306 %X Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. Based on this perspective, we propose $\textbf{TAC}$, a Transformer-based audio tokenizer that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction of general audio. We show that a simple, fully end-to-end learned tokenizer built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, the proposed tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging TAC’s discrete tokens, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, TAC enables competitive ASR performance without auxiliary encoders. Our findings position TAC as a unified, scalable interface for the next generation of native audio foundation models.
APA
Gong, Y., Chen, K., Fei, Z., Yang, X., Chen, K., Wang, Y., Huang, K., Chen, M., Li, R., Cheng, Q., Li, S. & Qiu, X.. (2026). Scaling Transformers for End-to-End Discrete Audio Tokenization. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:35903-35924 Available from https://proceedings.mlr.press/v306/gong26c.html.

Related Material