DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, Qifan Song
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:2341-2349, 2026.

Abstract

Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose DynamIc VErification RElaxed SpEculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED: learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: \url{https://github.com/comeusr/diversed}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-wang26d, title = { DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification }, author = {Wang, Ziyi and Kasa, Siva Rajesh and S, Ankith M and Kasa, Santhosh Kumar and Zou, Jiaru and Negi, Sumit and Zhang, Ruqi and Jiang, Nan and Song, Qifan}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {2341--2349}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/wang26d/wang26d.pdf}, url = {https://proceedings.mlr.press/v300/wang26d.html}, abstract = { Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose DynamIc VErification RElaxed SpEculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED: learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: \url{https://github.com/comeusr/diversed}. } }
Endnote
%0 Conference Paper %T DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification %A Ziyi Wang %A Siva Rajesh Kasa %A Ankith M S %A Santhosh Kumar Kasa %A Jiaru Zou %A Sumit Negi %A Ruqi Zhang %A Nan Jiang %A Qifan Song %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-wang26d %I PMLR %P 2341--2349 %U https://proceedings.mlr.press/v300/wang26d.html %V 300 %X Speculative decoding is an effective technique for accelerating large language model inference by drafting multiple tokens in parallel. In practice, its speedup is often bottlenecked by a rigid verification step that strictly enforces the accepted token distribution to exactly match the target model. This constraint leads to the rejection of many plausible tokens, lowering the acceptance rate and limiting overall time speedup. To overcome this limitation, we propose DynamIc VErification RElaxed SpEculative Decoding (DIVERSED), a relaxed verification framework that improves time efficiency while preserving generation quality. DIVERSED: learns an ensemble-based verifier that blends the draft and target model distributions with a task-dependent and context-dependent weight. We provide theoretical justification for our approach and demonstrate empirically that DIVERSED achieves substantially higher inference efficiency compared to standard speculative decoding methods. Code is available at: \url{https://github.com/comeusr/diversed}.
APA
Wang, Z., Kasa, S.R., S, A.M., Kasa, S.K., Zou, J., Negi, S., Zhang, R., Jiang, N. & Song, Q.. (2026). DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:2341-2349 Available from https://proceedings.mlr.press/v300/wang26d.html.

Related Material