DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark D. Hill, Murali Annavaram
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:33270-33284, 2026.

Abstract

Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades Time-Between-Tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives. DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU–GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3$\times$ while maintaining low generation latency compared to state-of-the-art frameworks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gao26m, title = {{D}uet{S}erve: Harmonizing Prefill and Decode for {LLM} Serving via Adaptive {GPU} Multiplexing}, author = {Gao, Lei and Jiang, Chaoyi and Zarch, Hossein Entezari and Wong, Daniel and Hill, Mark D. and Annavaram, Murali}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {33270--33284}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gao26m/gao26m.pdf}, url = {https://proceedings.mlr.press/v306/gao26m.html}, abstract = {Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades Time-Between-Tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives. DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU–GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3$\times$ while maintaining low generation latency compared to state-of-the-art frameworks.} }
Endnote
%0 Conference Paper %T DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing %A Lei Gao %A Chaoyi Jiang %A Hossein Entezari Zarch %A Daniel Wong %A Mark D. Hill %A Murali Annavaram %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gao26m %I PMLR %P 33270--33284 %U https://proceedings.mlr.press/v306/gao26m.html %V 306 %X Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades Time-Between-Tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives. DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU–GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3$\times$ while maintaining low generation latency compared to state-of-the-art frameworks.
APA
Gao, L., Jiang, C., Zarch, H.E., Wong, D., Hill, M.D. & Annavaram, M.. (2026). DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:33270-33284 Available from https://proceedings.mlr.press/v306/gao26m.html.

Related Material