Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding

Marianne Arriola, Volodymyr Kuleshov
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:3819-3855, 2026.

Abstract

Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by unmasking token blocks left-to-right, but it is still limited to generate fixed-size blocks sequentially. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/setdlms/

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-arriola26a, title = {Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding}, author = {Arriola, Marianne and Kuleshov, Volodymyr}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {3819--3855}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/arriola26a/arriola26a.pdf}, url = {https://proceedings.mlr.press/v306/arriola26a.html}, abstract = {Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by unmasking token blocks left-to-right, but it is still limited to generate fixed-size blocks sequentially. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/setdlms/} }
Endnote
%0 Conference Paper %T Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding %A Marianne Arriola %A Volodymyr Kuleshov %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-arriola26a %I PMLR %P 3819--3855 %U https://proceedings.mlr.press/v306/arriola26a.html %V 306 %X Discrete diffusion models have steadily improved in quality relative to autoregressive (AR) models. However, these models are normally constrained to fixed-length generation and do not support key-value (KV) caching. Block diffusion partially bridges diffusion and AR by unmasking token blocks left-to-right, but it is still limited to generate fixed-size blocks sequentially. Here, we present a new class of language models, set diffusion, comprised of (i) a likelihood parameterization that factorizes over flexible-position, flexible-length token sets and (ii) a set-causal diffusion architecture that supports KV cache updates after every inference step. By factorizing over token sets instead of fixed-size blocks, tokens can be decoded in arbitrarily-ordered sets, including sliding-window sets, enabling faster inference and support for any-order decoding. Set diffusion achieves better speed-quality tradeoffs on mathematical reasoning, summarization, and unconditional generation compared to prior diffusion language models while offering stronger infilling performance than block diffusion. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/setdlms/
APA
Arriola, M. & Kuleshov, V.. (2026). Set Diffusion: Interpolating Token Orderings between Autoregression and Diffusion for Fast and Flexible Decoding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:3819-3855 Available from https://proceedings.mlr.press/v306/arriola26a.html.

Related Material