Autoregression with Self-Token Prediction

Dengsheng Chen, Yangming Shi, Enhua Wu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:16630-16656, 2026.

Abstract

Conventional autoregressive models achieve causality through next-token prediction, but suffer from prohibitive latency and typically underperform non-causal alternatives such as masked prediction and diffusion. We propose self-token prediction, which enables predicting a flexible number of tokens per step, and introduce SAR, the first spatially autoregressive image generator built on this paradigm. SAR delivers markedly faster inference speeds and consistently outperforms prior autoregressive baselines, achieving performance on par with state-of-the-art non-causal models. Our findings highlight self-token prediction as a crucial step toward a high-quality autoregressive paradigm for visual generation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26du, title = {Autoregression with Self-Token Prediction}, author = {Chen, Dengsheng and Shi, Yangming and Wu, Enhua}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {16630--16656}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26du/chen26du.pdf}, url = {https://proceedings.mlr.press/v306/chen26du.html}, abstract = {Conventional autoregressive models achieve causality through next-token prediction, but suffer from prohibitive latency and typically underperform non-causal alternatives such as masked prediction and diffusion. We propose self-token prediction, which enables predicting a flexible number of tokens per step, and introduce SAR, the first spatially autoregressive image generator built on this paradigm. SAR delivers markedly faster inference speeds and consistently outperforms prior autoregressive baselines, achieving performance on par with state-of-the-art non-causal models. Our findings highlight self-token prediction as a crucial step toward a high-quality autoregressive paradigm for visual generation.} }
Endnote
%0 Conference Paper %T Autoregression with Self-Token Prediction %A Dengsheng Chen %A Yangming Shi %A Enhua Wu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26du %I PMLR %P 16630--16656 %U https://proceedings.mlr.press/v306/chen26du.html %V 306 %X Conventional autoregressive models achieve causality through next-token prediction, but suffer from prohibitive latency and typically underperform non-causal alternatives such as masked prediction and diffusion. We propose self-token prediction, which enables predicting a flexible number of tokens per step, and introduce SAR, the first spatially autoregressive image generator built on this paradigm. SAR delivers markedly faster inference speeds and consistently outperforms prior autoregressive baselines, achieving performance on par with state-of-the-art non-causal models. Our findings highlight self-token prediction as a crucial step toward a high-quality autoregressive paradigm for visual generation.
APA
Chen, D., Shi, Y. & Wu, E.. (2026). Autoregression with Self-Token Prediction. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:16630-16656 Available from https://proceedings.mlr.press/v306/chen26du.html.

Related Material