EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

Minsoo Kim, Arnav Kundu, Han-Byul Kim, Richa Dixit, Minsik Cho
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:58057-58078, 2026.

Abstract

Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near-full-cache accuracy under $4$–$6\times$ compression, and reduces latency and peak memory by up to $2.4\times$ and $3.7\times$, respectively.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-kim26ag, title = {{E}pi{C}ache: Episodic {KV} Cache Management for Long-Term Conversation on Resource-Constrained Environments}, author = {Kim, Minsoo and Kundu, Arnav and Kim, Han-Byul and Dixit, Richa and Cho, Minsik}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {58057--58078}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/kim26ag/kim26ag.pdf}, url = {https://proceedings.mlr.press/v306/kim26ag.html}, abstract = {Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near-full-cache accuracy under $4$–$6\times$ compression, and reduces latency and peak memory by up to $2.4\times$ and $3.7\times$, respectively.} }
Endnote
%0 Conference Paper %T EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments %A Minsoo Kim %A Arnav Kundu %A Han-Byul Kim %A Richa Dixit %A Minsik Cho %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-kim26ag %I PMLR %P 58057--58078 %U https://proceedings.mlr.press/v306/kim26ag.html %V 306 %X Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30%, achieves near-full-cache accuracy under $4$–$6\times$ compression, and reduces latency and peak memory by up to $2.4\times$ and $3.7\times$, respectively.
APA
Kim, M., Kundu, A., Kim, H., Dixit, R. & Cho, M.. (2026). EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:58057-58078 Available from https://proceedings.mlr.press/v306/kim26ag.html.

Related Material