KV cache offloading: what it actually costs you at each layer

KV cache offloading gets pitched as a free lunch: keep the working set on GPU HBM, push the rest to CPU RAM or a remote cache tier, and serve longer contexts at higher concurrency without buying more accelerators. It works, but the cost doesn't disappear — it moves to a different layer of the stack, and where it lands depends on your setup.

If you offload to CPU memory

This is the cheapest option and the one most serving frameworks support natively. The cost shows up as PCIe bandwidth pressure during cache swap-in, which turns into added latency on cache misses — worse on multi-tenant GPUs where PCIe is already shared across concurrent requests. It's a reasonable default for single-node deployments where you control the whole request path.

If you offload to a remote KV cache tier

Distributed caching (used by some gateway and orchestration layers to share cache across replicas) removes the single-node ceiling entirely, but now the cost is network — you've added a hop that has to beat the cost of just recomputing the cache from scratch. Whether that trade is worth it depends entirely on your prefix reuse rate: high reuse (agentic workloads, RAG with shared documents) makes remote caching a clear win; low reuse (mostly unique single-turn prompts) means you're paying network cost for cache entries that will never be read again.

If you don't offload at all

Keeping everything in HBM caps your effective context length and concurrency at whatever fits in GPU memory — the cost here is upfront and visible (you provision more or bigger accelerators) rather than showing up later as latency variance. For predictable-latency workloads (voice, real-time agents) this is often the right trade: a consistent, provisioned cost beats a cheaper-on-average one with a long tail.

Where this fits in the stack

Offloading strategy is decided at the inference framework and serving/orchestration layers, but it changes what you actually need from the accelerator hardware underneath — see how framework choice propagates through to instance selection in the deployment graph.