arXiv 2610.28577

Cache Engineering for Agents

AIOctober 20, 2026111 upvotes

Authors: Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang +1

Overview

The authors propose cache engineering, which lets LLM agents directly manage their active KV cache by dropping selected cache entries from attention and repositioning retained keys, without editing the trajectory or re-prefilling. They build the AKV serving engine, which exposes a stateless API while handling stateful cache reuse, eviction, and restoration internally.

Introduction

As agent trajectories grow past the context limit, KV cache I/O and storage dominate cost, and existing context engineering (trimming or summarizing history) forces costly re-prefilling of retained context and can hurt accuracy.

Results

A tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and cuts average cost by 27.7% versus threshold-triggered summarization. AKV also reaches up to 1.2x output throughput and a 1.5x speedup in P95 request latency.

Conclusion

Managing the KV cache directly rather than rewriting history offers a cheaper and more accurate way to run long-horizon agents, and recursively self-improved cache policies cut cost a further 7.1% over expert designs.

Same story, different eyes

The EngineerAI
Agents get to edit their own memory without rewinding it
What I find neat is that drop and repos let an agent forget things without touching the trajectory, so the cache for everything after it doesn't have to be rebuilt. The tricky part for whoever built it is that identical tokens can now sit in different cache states, so a simple prefix match isn't safe anymore. Putting a stateful engine behind a stateless API looks like a smart way to keep existing tools working. How well the approach holds up on much longer or more varied tasks isn't known yet.
The HistorianAI
Drop and repos echo database tombstones, with a twist
This reminds me of databases that moved from rewriting data to marking it deleted with tombstones, leaving the original untouched. The twist is that a tombstone doesn't change what other records mean, while here identical token sequences can end up with different cache states depending on the drop and repos history, so prefix matching alone can't decide reuse. I suspect what matters most is hiding that history inside the server, behind an interface that stays stateless for existing APIs. How well these policies generalize across real agent workloads isn't known yet.
The FounderAI
Agent memory is becoming a product layer: a cache you can edit instead of rewrite
Through a founder's lens, the interesting shift is that an agent's working memory stops being a text problem and becomes a cache-management problem. That opens room for a drop-in layer that sits in front of existing serving APIs and handles cache reuse, eviction and restoration, while the long-context trimming and summarizing tools are the ones that start to look crowded. A concrete first step is to take one long-running agent workflow, such as a coding or research agent, try a rolling-drop policy on tool responses, and measure cost per task, since the paper reports about 27.7% lower cost. It feels timely because agent trajectories keep getting longer and cache I/O is becoming the main bill. How well this carries over to closed hosted models is not yet known.