Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

Wenhung Lee, Jian-Jia Chen, Xiaolin Lin, Pei-Shuo Wang, Chi-Chih Chang, Chun-Che Yang, Wei-Chen Wang, Hanrui Wang, Ning-Chi Huang, Grace Li Zhang, Kai-Chiang Wu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:63471-63495, 2026.

Abstract

While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85$\times$ speedup in self-attention and a 9.17$\times$ end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-lee26d, title = {Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding}, author = {Lee, Wenhung and Chen, Jian-Jia and Lin, Xiaolin and Wang, Pei-Shuo and Chang, Chi-Chih and Yang, Chun-Che and Wang, Wei-Chen and Wang, Hanrui and Huang, Ning-Chi and Zhang, Grace Li and Wu, Kai-Chiang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {63471--63495}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/lee26d/lee26d.pdf}, url = {https://proceedings.mlr.press/v306/lee26d.html}, abstract = {While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85$\times$ speedup in self-attention and a 9.17$\times$ end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.} }
Endnote
%0 Conference Paper %T Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding %A Wenhung Lee %A Jian-Jia Chen %A Xiaolin Lin %A Pei-Shuo Wang %A Chi-Chih Chang %A Chun-Che Yang %A Wei-Chen Wang %A Hanrui Wang %A Ning-Chi Huang %A Grace Li Zhang %A Kai-Chiang Wu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-lee26d %I PMLR %P 63471--63495 %U https://proceedings.mlr.press/v306/lee26d.html %V 306 %X While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85$\times$ speedup in self-attention and a 9.17$\times$ end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.
APA
Lee, W., Chen, J., Lin, X., Wang, P., Chang, C., Yang, C., Wang, W., Wang, H., Huang, N., Zhang, G.L. & Wu, K.. (2026). Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:63471-63495 Available from https://proceedings.mlr.press/v306/lee26d.html.

Related Material