Ekka: Automated Diagnosis of Silent Errors in LLM Inference

Yile Gu, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, Baris Kasikci
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:37360-37375, 2026.

Abstract

LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80% pass@$1$ diagnosis accuracy and 88% pass@$5$ diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gu26q, title = {Ekka: Automated Diagnosis of Silent Errors in {LLM} Inference}, author = {Gu, Yile and Zhang, Zhen and Zhu, Shaowei and Fu, Xinwei and Wu, Jun and Wang, Yida and Kasikci, Baris}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {37360--37375}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gu26q/gu26q.pdf}, url = {https://proceedings.mlr.press/v306/gu26q.html}, abstract = {LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80% pass@$1$ diagnosis accuracy and 88% pass@$5$ diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.} }
Endnote
%0 Conference Paper %T Ekka: Automated Diagnosis of Silent Errors in LLM Inference %A Yile Gu %A Zhen Zhang %A Shaowei Zhu %A Xinwei Fu %A Jun Wu %A Yida Wang %A Baris Kasikci %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gu26q %I PMLR %P 37360--37375 %U https://proceedings.mlr.press/v306/gu26q.html %V 306 %X LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output quality silently degrades without any explicit error signals. Diagnosing silent errors is notoriously difficult due to the substantial semantic gap between the high-level symptoms and the low-level root causes. We observe that diagnosis of silent errors can be effectively framed as a differential debugging problem by leveraging the existence of semantically correct reference implementations. We propose Ekka, an automated diagnosis system that identifies root causes by systematically aligning and comparing intermediate execution states between a target and a reference framework. We constructed a benchmark of real-world silent errors from popular serving frameworks, where Ekka shows 80% pass@$1$ diagnosis accuracy and 88% pass@$5$ diagnosis accuracy, outperforming state-of-the-art systems. Ekka also diagnoses 4 new silent errors from serving frameworks, all of which have been confirmed by the developers.
APA
Gu, Y., Zhang, Z., Zhu, S., Fu, X., Wu, J., Wang, Y. & Kasikci, B.. (2026). Ekka: Automated Diagnosis of Silent Errors in LLM Inference. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:37360-37375 Available from https://proceedings.mlr.press/v306/gu26q.html.

Related Material