Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

Costin-Andrei Oncescu, Qingyang Wu, Wai Tong Chung, Tsai-Chuan Wu, Bryan Dev Gopal, Junxiong Wang, Tri Dao, Ben Athiwaratkun
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:94656-94674, 2026.

Abstract

An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for $\textbf{dynamically}$ re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a $\textbf{batch-aware routing}$ that works by having tokens $\textbf{piggyback}$ experts that have already been loaded into memory due to being crucial to other tokens within the same batch. At batch size $16$, OEA reduces MoE-layer decode latency by $39%$ on Qwen3-30B while preserving standard-error-adjusted downstream accuracy, and by $15%$ on Qwen3-235B with only small overall degradation on the long-generation benchmark suite.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-oncescu26a, title = {Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining}, author = {Oncescu, Costin-Andrei and Wu, Qingyang and Chung, Wai Tong and Wu, Tsai-Chuan and Gopal, Bryan Dev and Wang, Junxiong and Dao, Tri and Athiwaratkun, Ben}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {94656--94674}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/oncescu26a/oncescu26a.pdf}, url = {https://proceedings.mlr.press/v306/oncescu26a.html}, abstract = {An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for $\textbf{dynamically}$ re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a $\textbf{batch-aware routing}$ that works by having tokens $\textbf{piggyback}$ experts that have already been loaded into memory due to being crucial to other tokens within the same batch. At batch size $16$, OEA reduces MoE-layer decode latency by $39%$ on Qwen3-30B while preserving standard-error-adjusted downstream accuracy, and by $15%$ on Qwen3-235B with only small overall degradation on the long-generation benchmark suite.} }
Endnote
%0 Conference Paper %T Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining %A Costin-Andrei Oncescu %A Qingyang Wu %A Wai Tong Chung %A Tsai-Chuan Wu %A Bryan Dev Gopal %A Junxiong Wang %A Tri Dao %A Ben Athiwaratkun %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-oncescu26a %I PMLR %P 94656--94674 %U https://proceedings.mlr.press/v306/oncescu26a.html %V 306 %X An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for $\textbf{dynamically}$ re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a $\textbf{batch-aware routing}$ that works by having tokens $\textbf{piggyback}$ experts that have already been loaded into memory due to being crucial to other tokens within the same batch. At batch size $16$, OEA reduces MoE-layer decode latency by $39%$ on Qwen3-30B while preserving standard-error-adjusted downstream accuracy, and by $15%$ on Qwen3-235B with only small overall degradation on the long-generation benchmark suite.
APA
Oncescu, C., Wu, Q., Chung, W.T., Wu, T., Gopal, B.D., Wang, J., Dao, T. & Athiwaratkun, B.. (2026). Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:94656-94674 Available from https://proceedings.mlr.press/v306/oncescu26a.html.

Related Material