DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

Vaibhav Singh, Oleksiy Ostapenko, Pierre-Andre Noel, Eugene Belilovsky, Torsten Scholak
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:113484-113504, 2026.

Abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2$\times$ and 4.3$\times$ higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-singh26d, title = {{D}iffu{M}amba: High-Throughput Diffusion {LM}s with Mamba Backbone}, author = {Singh, Vaibhav and Ostapenko, Oleksiy and Noel, Pierre-Andre and Belilovsky, Eugene and Scholak, Torsten}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {113484--113504}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/singh26d/singh26d.pdf}, url = {https://proceedings.mlr.press/v306/singh26d.html}, abstract = {Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2$\times$ and 4.3$\times$ higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.} }
Endnote
%0 Conference Paper %T DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone %A Vaibhav Singh %A Oleksiy Ostapenko %A Pierre-Andre Noel %A Eugene Belilovsky %A Torsten Scholak %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-singh26d %I PMLR %P 113484--113504 %U https://proceedings.mlr.press/v306/singh26d.html %V 306 %X Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency due to quadratic attention or KV-cache overhead. We introduce DiffuMamba, a masked diffusion language model built on a bidirectional Mamba backbone that combines the diffusion objective with linear-time sequence modeling, and DiffuMamba-H, a hybrid variant with interleaved attention. Across scales up to 1.3B parameters, our models match Transformer-based diffusion in downstream performance while achieving up to 8.2$\times$ and 4.3$\times$ higher inference throughput, respectively, on long sequences. We further present a systematic analysis of inference efficiency across modern DLM variants, combining asymptotic complexity with empirical measurements. Notably, cache-efficient block diffusion with Mamba mixers emerges as the only strategy that scales linearly with sequence length and achieves the strongest performance across all baselines, suggesting a promising direction for future diffusion-based generation systems.
APA
Singh, V., Ostapenko, O., Noel, P., Belilovsky, E. & Scholak, T.. (2026). DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:113484-113504 Available from https://proceedings.mlr.press/v306/singh26d.html.

Related Material