SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning

Jun Liu, Peilin Liu, Ruicheng Zhang, Zhang Senlei, Yanbo Chen, Ziao Wang, Jinyun Yang, Mingqi Wang, Shuhao Zhang, Xiaofei Liao, Hai Jin
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:77613-77638, 2026.

Abstract

Large Language Model (LLM) applications increasingly execute as end-to-end inference pipelines that couple generation with retrieval, stateful memory, context refinement, and tool use under strict tail-latency and Service-Level Objective (SLO) constraints. Today, these stages are often stitched together as RPC-connected services, obscuring cross-stage queueing and interference and limiting pipeline-level compilation and resource sharing. We present SAGE (Streaming-Augmented Generative Execution), a full-stack system that treats inference pipelines as first-class compilation targets. SAGE exposes pipelines as declarative dataflows and compiles them into distributed execution plans with bounded-queue backpressure. It integrates vector search, streaming semantic state, structured memory, and refinement as operators with explicit resource/state contracts, enabling operator-level diagnosis of tail behavior. SAGE integrates pluggable generation and embedding backends and provides a unified control plane for engine management, batching, and admission under mixed workloads. On a 16-node cluster, SAGE sustains 16 requests/s at $>700$ tokens/request with 1 ms median scheduling overhead, and achieves near-linear scale-out to 16 nodes (11.4$\times$ throughput at 16 nodes), and reduces P99 latency by 57% under multi-pipeline contention versus simultaneous admission.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-liu26co, title = {{SAGE}: A Dataflow-Native Framework for Modular, Controllable, and Transparent {LLM}-Augmented Reasoning}, author = {Liu, Jun and Liu, Peilin and Zhang, Ruicheng and Senlei, Zhang and Chen, Yanbo and Wang, Ziao and Yang, Jinyun and Wang, Mingqi and Zhang, Shuhao and Liao, Xiaofei and Jin, Hai}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {77613--77638}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liu26co/liu26co.pdf}, url = {https://proceedings.mlr.press/v306/liu26co.html}, abstract = {Large Language Model (LLM) applications increasingly execute as end-to-end inference pipelines that couple generation with retrieval, stateful memory, context refinement, and tool use under strict tail-latency and Service-Level Objective (SLO) constraints. Today, these stages are often stitched together as RPC-connected services, obscuring cross-stage queueing and interference and limiting pipeline-level compilation and resource sharing. We present SAGE (Streaming-Augmented Generative Execution), a full-stack system that treats inference pipelines as first-class compilation targets. SAGE exposes pipelines as declarative dataflows and compiles them into distributed execution plans with bounded-queue backpressure. It integrates vector search, streaming semantic state, structured memory, and refinement as operators with explicit resource/state contracts, enabling operator-level diagnosis of tail behavior. SAGE integrates pluggable generation and embedding backends and provides a unified control plane for engine management, batching, and admission under mixed workloads. On a 16-node cluster, SAGE sustains 16 requests/s at $>700$ tokens/request with 1 ms median scheduling overhead, and achieves near-linear scale-out to 16 nodes (11.4$\times$ throughput at 16 nodes), and reduces P99 latency by 57% under multi-pipeline contention versus simultaneous admission.} }
Endnote
%0 Conference Paper %T SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning %A Jun Liu %A Peilin Liu %A Ruicheng Zhang %A Zhang Senlei %A Yanbo Chen %A Ziao Wang %A Jinyun Yang %A Mingqi Wang %A Shuhao Zhang %A Xiaofei Liao %A Hai Jin %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liu26co %I PMLR %P 77613--77638 %U https://proceedings.mlr.press/v306/liu26co.html %V 306 %X Large Language Model (LLM) applications increasingly execute as end-to-end inference pipelines that couple generation with retrieval, stateful memory, context refinement, and tool use under strict tail-latency and Service-Level Objective (SLO) constraints. Today, these stages are often stitched together as RPC-connected services, obscuring cross-stage queueing and interference and limiting pipeline-level compilation and resource sharing. We present SAGE (Streaming-Augmented Generative Execution), a full-stack system that treats inference pipelines as first-class compilation targets. SAGE exposes pipelines as declarative dataflows and compiles them into distributed execution plans with bounded-queue backpressure. It integrates vector search, streaming semantic state, structured memory, and refinement as operators with explicit resource/state contracts, enabling operator-level diagnosis of tail behavior. SAGE integrates pluggable generation and embedding backends and provides a unified control plane for engine management, batching, and admission under mixed workloads. On a 16-node cluster, SAGE sustains 16 requests/s at $>700$ tokens/request with 1 ms median scheduling overhead, and achieves near-linear scale-out to 16 nodes (11.4$\times$ throughput at 16 nodes), and reduces P99 latency by 57% under multi-pipeline contention versus simultaneous admission.
APA
Liu, J., Liu, P., Zhang, R., Senlei, Z., Chen, Y., Wang, Z., Yang, J., Wang, M., Zhang, S., Liao, X. & Jin, H.. (2026). SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:77613-77638 Available from https://proceedings.mlr.press/v306/liu26co.html.

Related Material