ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

Xinyi. Hu, Yuhao Shen, Zhang Baolin, Hengxin Zhang, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:45636-45656, 2026.

Abstract

Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20% relative speedup gain against the strongest baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-hu26ar, title = {{ECHO}: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios}, author = {Hu, Xinyi. and Shen, Yuhao and Baolin, Zhang and Zhang, Hengxin and Dai, Jun and Ge, Shuang and Lei, Chen and Li, Yue and Wan, Mingcheng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {45636--45656}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hu26ar/hu26ar.pdf}, url = {https://proceedings.mlr.press/v306/hu26ar.html}, abstract = {Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20% relative speedup gain against the strongest baselines.} }
Endnote
%0 Conference Paper %T ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios %A Xinyi. Hu %A Yuhao Shen %A Zhang Baolin %A Hengxin Zhang %A Jun Dai %A Shuang Ge %A Chen Lei %A Yue Li %A Mingcheng Wan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hu26ar %I PMLR %P 45636--45656 %U https://proceedings.mlr.press/v306/hu26ar.html %V 306 %X Speculative Decodin promises to accelerate Large Language Model inference, yet its efficacy often degrades in production-grade scenarios. Existing evaluations typically overlook the compute-bound nature of high-concurrency regimes, where verification compute becomes the dominant bottleneck. Consequently, prior methods face a dilemma: static trees incur massive verification waste, while dynamic trees suffer from cumulative misjudgments and kernel incompatibility. To bridge this gap, we introduce ECHO, a high concurrency-oriented framework integrated into SGLang that reformulates speculative execution as a budgeted scheduling problem. Crucially, ECHO employs sparse confidence gating to manage the batch as a unified super-tree, elastically pivoting budget between depth and width to co-optimize the trade-off between reducing global verification steps and maximizing per-step efficiency. Extensive evaluations across diverse model scales—particularly the industrial-grade Qwen3-235B—demonstrate that ECHO consistently outperforms state-of-the-art baselines in both low-load and high-load scenarios, achieving up to 5.35$\times$ walltime speedup and delivering over 20% relative speedup gain against the strongest baselines.
APA
Hu, X., Shen, Y., Baolin, Z., Zhang, H., Dai, J., Ge, S., Lei, C., Li, Y. & Wan, M.. (2026). ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:45636-45656 Available from https://proceedings.mlr.press/v306/hu26ar.html.

Related Material